eval/chat_sim/README.md

# Chat simulator (end-to-end, in-process)

Drives the **real** `ChatHandler.handle()` in-process to simulate a user chatting in
one analysis session — from creation through a report — and prints **what each step
does**: router decision, slow-path status, every tool call, every LLM call (output +
tokens + latency), the streamed answer, and a final report built from the run's
`report_inputs`.

In-process (not via the HTTP server) on purpose: the SSE endpoint hides the internal
LLM outputs (router/planner/assembler), so a script that consumes `/chat/stream` can't
show them. This hooks the same tracer seam the handler threads its Langfuse callbacks
+ tool spans through (`ScriptTracer.active=True`) — **no source changes**.

## Run

Module mode (`-m`) so `src` imports resolve; needs a populated `.env` (Azure OpenAI +
Postgres + Azure Blob for the Titanic Parquet). `ENABLE_SLOW_PATH` is forced on here.

```bash
uv run python -m eval.chat_sim.run_chat                  # scripted Titanic convo + report
uv run python -m eval.chat_sim.run_chat --interactive    # type your own messages; 'report' / 'exit'
uv run python -m eval.chat_sim.run_chat --max-turns 1 --no-report   # cheap smoke test
uv run python -m eval.chat_sim.run_chat --no-bind        # planner sees the whole catalog (not just Titanic)
uv run python -m eval.chat_sim.run_chat --plain          # no ANSI colors
```

(Or `./.venv/Scripts/python.exe -m eval.chat_sim.run_chat ...` on Windows.)

## What it does each run

1. **Creates a fresh analysis** (`AnalysisStateStore.create`) with an objective, and
   **scopes it to the Titanic source** by seeding an analysis-scope `data_catalog` row
   (the user catalog restricted to Titanic) so `structured_flow` is scoped to one
   source — same as `/analysis/create` (in production Go materializes this row from
   `analyses.data_bind`).
2. **Runs each turn** through `handle()` and prints the router decision, slow-path
   status pings, the tool table (kind / rows / latency / error), the LLM table
   (in/out/total tokens + ms + a prompt/output snippet), and the answer + sources.
3. **Generates a report** (mirrors `POST /report`: floor check → `ReportGenerator` →
   `ReportStore`) from the `report_inputs` the structured turns persisted.

## Notes

- **Default user** is `4b5d1bac-…` whose playground catalog has the Titanic CSV
  (tabular) + a "dummy" Postgres (schema). Override with `--user-id`.
- **Writes to the DB the `.env` points at** — a fresh `analyses` row + `report_inputs`
  + a `reports` row per run. Point `.env` at the playground DB. Each run is a new
  `analysis_id` (printed at the end) so runs don't collide.
- **"output: <no text content — structured / tool-call output>"** is expected for the
  router, planner, and assembler — they use structured/function-call output, so the
  LLM message has no plain text. Their result shows up in the ROUTER line, the plan,
  and the streamed ANSWER respectively. The chatbot/help calls stream text, so their
  output is shown (annotated `masked→cloud` where the cloud trace would redact it).
- The first scripted question may route to `chat` rather than `help` — that's the live
  router's real call, shown transparently.