eval/chat_sim/README.md

Chat simulator (end-to-end, in-process)

Drives the real ChatHandler.handle() in-process to simulate a user chatting in one analysis session — from creation through a report — and prints what each step does: router decision, slow-path status, every tool call, every LLM call (output + tokens + latency), the streamed answer, and a final report built from the run's report_inputs.

In-process (not via the HTTP server) on purpose: the SSE endpoint hides the internal LLM outputs (router/planner/assembler), so a script that consumes /chat/stream can't show them. This hooks the same tracer seam the handler threads its Langfuse callbacks

• tool spans through (ScriptTracer.active=True) — no source changes.

Run

Module mode (-m) so src imports resolve; needs a populated .env (Azure OpenAI + Postgres + Azure Blob for the Titanic Parquet). ENABLE_SLOW_PATH is forced on here.

uv run python -m eval.chat_sim.run_chat                  # scripted Titanic convo + report
uv run python -m eval.chat_sim.run_chat --interactive    # type your own messages; 'report' / 'exit'
uv run python -m eval.chat_sim.run_chat --max-turns 1 --no-report   # cheap smoke test
uv run python -m eval.chat_sim.run_chat --no-bind        # planner sees the whole catalog (not just Titanic)
uv run python -m eval.chat_sim.run_chat --plain          # no ANSI colors

(Or ./.venv/Scripts/python.exe -m eval.chat_sim.run_chat ... on Windows.)

What it does each run

1. Creates a fresh analysis (AnalysisStateStore.create) with an objective, and scopes it to the Titanic source by seeding an analysis-scope data_catalog row (the user catalog restricted to Titanic) so structured_flow is scoped to one source — same as /analysis/create (in production Go materializes this row from analyses.data_bind).
2. Runs each turn through handle() and prints the router decision, slow-path status pings, the tool table (kind / rows / latency / error), the LLM table (in/out/total tokens + ms + a prompt/output snippet), and the answer + sources.
3. Generates a report (mirrors POST /report: floor check → ReportGenerator → ReportStore) from the report_inputs the structured turns persisted.

Notes

• Default user is 4b5d1bac-… whose playground catalog has the Titanic CSV (tabular) + a "dummy" Postgres (schema). Override with --user-id.
• Writes to the DB the .env points at — a fresh analyses row + report_inputs
  • a reports row per run. Point .env at the playground DB. Each run is a new analysis_id (printed at the end) so runs don't collide.
• "output: <no text content — structured / tool-call output>" is expected for the router, planner, and assembler — they use structured/function-call output, so the LLM message has no plain text. Their result shows up in the ROUTER line, the plan, and the streamed ANSWER respectively. The chatbot/help calls stream text, so their output is shown (annotated masked→cloud where the cloud trace would redact it).
• The first scripted question may route to chat rather than help — that's the live router's real call, shown transparently.