docs/persistence.md

Persistence Design (HF /data now, Railway later)

This document describes a storage design that enables:

• Run history for all three modes (AI↔AI, Human↔AI, Upload Text): chronological list → click → replay read-only.
• Persona CRUD (create/update/delete) with versioning, while preserving historical run replay.
• Portability across deployment targets:
  • Hugging Face Spaces: persistent filesystem under /data
  • Future: Railway (likely Postgres), without rewriting app/business logic

This is intentionally implementation-agnostic at the interface level and implementation-specific only at the backend adapter level (SQLite first).

---

Goals

1. Durable run replay

   • After a run ends, the transcript + analysis outputs are persisted and can be reloaded after restart/redeploy.
   • Reloaded runs are read-only: no WebSocket streaming, no “resume”, no edits.

2. Stable historical fidelity

   • Reloading a run shows the same transcript and analysis output that was originally produced.
   • Runs do not drift if personas/system prompts are edited later.

3. One shared history

   • This app is currently used by a team evaluating the tool, so history is global/shared, not per-user.

4. Provider portability

   • HF → Railway should be a storage-backend swap (SQLite-on-volume → Postgres), not a rewrite.

---

Non-goals (for the first iteration)

• Live persistence during streaming (we can add later). For now: persist at end-of-run only.
• “Resume” or “continue” a prior run.
• Fine-grained multi-user auth and per-user histories.
• Full-text search, tagging, sharing links, etc.

---

What “persistent storage” means in this project

The current runtime holds critical data in memory during a session:

• Transcript messages: ConversationAI/backend/api/conversation_service.py maintains self.transcripts[conversation_id].
• Analysis results: after completion, _run_resource_agent() broadcasts resource_agent_result back to the UI.

Persistence means:

1. When a session ends as a sealed run (conversation finished and analysis succeeded), we write a Run record to durable storage:
   • transcript (messages)
   • analysis outputs (resource agent JSON, evidence catalog, schema versions)
   • configuration snapshot (LLM backend/model/params, selected personas, and the effective shared settings used)
   • persona snapshots for historical fidelity (see below)
2. We expose APIs to:
   • list prior runs (chronological)
   • fetch a specific run by run_id
3. The UI can then “rehydrate” a panel from those persisted artifacts and render it read-only.

---

Key design principles

1) Runs are immutable once sealed

We treat a “Run” as a record of what happened.

• A run can transition from “active” → “sealed”
• Once sealed, it is read-only for all consumers
• The UI is allowed to render it, export it, and inspect it

This makes the system easy to reason about and prevents accidental data drift.

2) Personas are mutable, but runs never drift

Personas and system prompts are editable over time (CRUD). However, old runs must still open exactly as they were.

To guarantee this:

• We store a persona snapshot inside each run (the persona content used at runtime).
• Optionally, we also store a reference to the persona ID/version that the snapshot came from.

3) Store blobs as JSON, but keep query fields as columns

For history lists and basic filtering, we want queryable columns (mode, status, timestamps). For richer data (config, analysis output, persona content), JSON is fine and reduces schema churn.

4) Storage adapter boundary (portability)

All application code should talk to a small interface (conceptually):

• RunStore
  • save_sealed_run(run_record)
  • list_runs(mode?, limit?, offset?)
  • get_run(run_id)
• PersonaStore
  • list_personas(kind?, include_deleted?)
  • get_persona(persona_id, version?)
  • create_persona(payload)
  • update_persona(persona_id, payload) (creates a new version)
  • delete_persona(persona_id) (soft delete)

HF and Railway differ only in the implementation of these interfaces.

---

Storage backend choice: SQLite-first

SQLite is selected as the first implementation because it provides:

• Atomic writes and durability
• Indexes for fast run listing
• Natural support for persona versioning
• A smooth conceptual migration path to Postgres later

Database location

• Hugging Face Spaces: DB_PATH=/data/converta/converta.db
• Local dev fallback: DB_PATH=ConversationAI/.localdata/converta.db (or similar)

The storage module should create parent directories if missing.

Important: DB_PATH (env) is the canonical source of truth for storage location. Any legacy config values (e.g. config/default_config.yaml SQLite path) should be treated as non-authoritative for persistence.

---

Proposed SQLite schema (v1)

runs — top-level history entries

• run_id TEXT PRIMARY KEY (UUID)
• mode TEXT NOT NULL (ai_to_ai|human_to_ai|text_analysis)
• status TEXT NOT NULL (completed in v1; reserve aborted|error for later)
• created_at TEXT NOT NULL (ISO timestamp)
• ended_at TEXT NOT NULL (ISO timestamp)
• title TEXT (optional)
• input_summary TEXT (optional: filename/source label for text analysis)
• config_json TEXT NOT NULL (JSON blob)
• sealed_at TEXT NOT NULL (ISO timestamp; equals ended_at in v1)

Indexes:

• INDEX runs_mode_created_at ON runs(mode, created_at DESC)
• INDEX runs_created_at ON runs(created_at DESC)

run_messages — transcripts

• run_id TEXT NOT NULL (FK → runs.run_id)
• message_index INTEGER NOT NULL
• role TEXT NOT NULL
• persona_label TEXT (optional)
• content TEXT NOT NULL
• timestamp TEXT (ISO timestamp)

Primary key:

• (run_id, message_index)

run_analyses — analysis outputs per run

• run_id TEXT NOT NULL (FK)
• analysis_key TEXT NOT NULL (e.g. resource_agent_v2)
• schema_version TEXT
• prompt_version TEXT
• result_json TEXT NOT NULL (full JSON blob, including evidence_catalog)

Primary key:

• (run_id, analysis_key)

personas — stable identity and lifecycle

• persona_id TEXT PRIMARY KEY (UUID)
• kind TEXT NOT NULL (surveyor|patient)
• name TEXT NOT NULL
• is_deleted INTEGER NOT NULL DEFAULT 0
• created_at TEXT NOT NULL
• updated_at TEXT NOT NULL

persona_versions — append-only versions

• persona_id TEXT NOT NULL (FK)
• version_id TEXT NOT NULL (UUID)
• created_at TEXT NOT NULL
• content_json TEXT NOT NULL (persona definition + system prompt)

Primary key:

• (persona_id, version_id)

run_persona_snapshots — prevent drift

• run_id TEXT NOT NULL (FK)
• role TEXT NOT NULL (surveyor|patient)
• persona_id TEXT (nullable)
• persona_version_id TEXT (nullable)
• snapshot_json TEXT NOT NULL

Primary key:

• (run_id, role)

---

What gets stored in config_json (recommended)

config_json should allow exact replay and debugging:

• LLM settings:
  • llm_backend, host, model
  • timeout, max_retries, retry_delay
  • any generation params used (temperature, max_tokens, top_p, etc.)
• Mode-specific:
  • AI↔AI: surveyor/patient persona IDs (and persona version ids / snapshots)
  • Human↔AI: same + human mode flags
  • Text analysis: source_name, optional file metadata (original filename, sha256)
• App versions:
  • analysis_prompt_version and schema_version (duplicated in run_analyses is fine)
  • optional git commit SHA (if available at runtime)

---

Integration points (where persistence is hooked in)

This section maps “what to save” to the current code.

1) AI↔AI and Human↔AI runs

Source of truth today:

• Transcript: ConversationAI/backend/api/conversation_service.py (self.transcripts[conversation_id])
• Analysis: _run_resource_agent(conversation_id) broadcasts resource_agent_result

End-of-run save flow (conceptual):

1. Run completes (or human chat ends)
2. Resource agent analysis completes successfully
3. Build a RunRecord:
   • run_id (use the conversation_id or generate a new UUID; recommended: new run_id distinct from WS id)
   • mode, status, timestamps
   • messages[] from self.transcripts
   • analyses["resource_agent_v2"] from the parsed JSON
   • persona snapshots for surveyor/patient content actually used
4. RunStore.save_sealed_run(run_record) writes the run to SQLite in a transaction.
5. Memory cleanup proceeds as today.

Notes:

• Because we’re “end-only”, if the process dies mid-run, that run is lost. This is accepted for v1.
• If a run is stopped/aborted (e.g. user presses Stop in AI↔AI), it is not a sealed run and is not persisted in v1.

2) Text analysis (“Upload Text”)

Source of truth today:

• Transcript is derived by parsing uploaded/pasted text into message-like units.
• Resource agent analysis is run and returned.

End-of-analysis save flow:

• Store as mode=text_analysis with messages[] using role transcript (or the derived roles if present).
• Store analysis output the same way as live conversations.
  • Because Upload Text is analysis-driven, “sealed” effectively means “analysis succeeded”; if analysis fails, do not persist.

---

API design for history and persona CRUD (v1)

Runs

• GET /api/runs?mode=ai_to_ai&limit=50&offset=0
  • Returns run summaries: run_id, mode, status, created_at, ended_at, title, input_summary
• GET /api/runs/{run_id}
  • Returns the full run record: transcript + analysis JSON + config snapshot

Personas

• GET /api/personas
• POST /api/personas
• PUT /api/personas/{persona_id} (creates a new version)
• DELETE /api/personas/{persona_id} (soft delete)
• (optional) GET /api/personas/{persona_id}/versions

Defaults:

• Default personas are seeded by the backend on startup.
• The DB is the runtime source of truth for both defaults and user-created personas.

---

UI behavior for read-only history

The UI should treat historical sessions as “render only”:

• No WebSocket connection
• Start/stop buttons disabled
• Transcript and analysis panels populated from GET /api/runs/{run_id}
• Export actions should be backed by server-canonical run data (avoid treating client-hydrated payloads as the source of truth)

Recommended UI structure:

• Add a “History” view per mode (or a unified history with filters)
• Clicking a run loads it into that panel, sets a readOnly=true flag, and renders accordingly

---

Migration plan: SQLite → Postgres (Railway)

What stays the same

• The RunStore / PersonaStore interface
• The external API endpoints and payload shapes
• The UI behavior (list + get + render)
• The logical schema (tables/entities)

What changes

• Replace SQLiteRunStore/SQLitePersonaStore with PostgresRunStore/PostgresPersonaStore
• DB connection config:
  • DATABASE_URL (Railway Postgres)
  • migrations managed via Alembic (recommended for Postgres)

Suggested migration path

1. Introduce the store interface now and keep it backend-agnostic.
2. Implement SQLite store now (HF + local).
3. When ready for Railway:
   • Implement Postgres store behind the same interface
   • Add migrations
   • Add a one-time “export/import” command:
     • export all SQLite runs/personas to JSON
     • import into Postgres

---

Verification checklist (for v1)

1. Run an AI↔AI session to completion and confirm a row appears in runs with status=completed.
2. Reload history via API and confirm:
   • transcript message count and ordering match the original UI
   • analysis boxes match the original output
3. Stop an AI↔AI session mid-run and confirm it does not appear in history (not sealed → not persisted).
4. Export a completed run via API and confirm the file downloads from /api/runs/{run_id}/export/* (server-canonical).
5. Restart the container/app and confirm history remains available.