docs/DSAGENT_DEV_GUIDE.md

# Lexsi DS Agent — Developer / KT Guide

A code-level companion to [`DSAGENT_OVERVIEW.md`](DSAGENT_OVERVIEW.md) (which is
the what/why). This is the **how**: file map, control flow, the contracts you'll
touch, configuration, and the full benchmark-running mechanics. Read the overview
first, then this.

---

## 1. Mental model in one paragraph

One **question** → one `AgentLoop.run()` → a **ReAct loop**: build a system prompt
(framing + active-dataset summary + tool catalog), ask the LLM for a JSON decision
(`{"thought","tool","args"}` or `{"thought","final_answer"}`), validate args with
the tool's pydantic schema, run the tool against a shared `AgentContext`, append the
observation to history, repeat until a final answer or the step budget. Tools read
and write a shared `ctx.cache` scratchpad. The data substrate is always **DuckDB**;
tabular ML runs on **remote Lexsi pods** via the SDK. The loop is generic — it has
no opinion on which tools to call.

---

## 2. Repo layout (code-oriented)

```
lexsi_ds/
  agent/
    loop.py            AgentLoop.run() — the ReAct orchestrator (start here)
    context.py         AgentContext, DatasetHandle, TableInfo, ColumnInfo
    prompts.py         PLANNER_SYSTEM, render_tool_catalog, tool-selector (intent routing)
    datasource.py      DuckDB handle loaders + multi-DB materialization + DB cache
    tools/
      base.py          Tool / ToolSpec / ToolResult contract (the interface)
      __init__.py      REGISTRY: dict[str, Tool]  (30 tools)
      <name>.py        one tool per file, each exports `TOOL: Tool`
      _tabular_common.py  shared helpers for the model-lifecycle tools
                          (resolve_tab_project, active_model_name, df_records, …)
    kg_index.py        embedding/lexical index over the PKDD knowledge graph
    table_index.py     embedding index over table schemas (schema linking, big DBs)
    session.py         SessionContext + multi-turn merge
    turn_classifier.py classify a turn: new / refine / abandon
    reference_resolver.py  resolve "it/that/those" against prior turns
    samples.py         curated demo questions (by_id, SAMPLES) — UI chips + tests
  llm/client.py        LLMClient protocol, StubLLMClient, LexsiTextClient, cache, timeout
  schema/
    raw.py             PKDD DDL introspection (introspect())
    context_graph.py   ContextGraph — the banking knowledge graph
  data/loader.py       PKDD TSVs -> artifacts/financial.duckdb
  paths.py             DUCKDB_PATH, ARTIFACTS_DIR, etc.

app/gradio_app.py      the UI (build_ui(); demo.launch on GRADIO_PORT)

benchmark/
  dab/
    adapter.py         parse a DAB dataset -> DatasetHandle + tasks (+ DB cache key)
    runner.py          run agent per DAB task, score Pass@1 via the task's validate.py
    DataAgentBench/    the (gitignored) upstream checkout + DB files
  pkdd_runner.py       run agent over bench/pkdd/questions.yaml, auto-score
  datagen/             template pipeline that GENERATES bench/pkdd/*.yaml

bench/pkdd/
  questions.yaml       40 single-turn bench items (analytic/predictive/adversarial)
  sessions.yaml        multi-turn sessions
  p2_feature_engineering.yaml   10 feature-eng items (separate `assertions` schema)
benchmark/fixtures/pkdd_seed.duckdb   pinned PKDD snapshot the bench scores against

scripts/
  repro_tabicl/                  TabICL inference + per-case XAI failure repro
  repro_model_lifecycle/         model-lifecycle SDK probe harness (self-contained):
    data/                        the PKDD CSVs (local copy)
    repro_model_lifecycle.py             TabICL-first (clean)
    repro_model_lifecycle_xgboost_first.py  XGBoost-first (#05-013 cascade repro)

tests/                 pytest suite (test_b_tools, test_v1_multiturn, test_dab_*, …)
docs/                  design + results + this guide
```

---

## 3. The run lifecycle ([loop.py](../lexsi_ds/agent/loop.py))

`AgentLoop(ctx, registry=REGISTRY, max_steps=18).run(question) -> AgentRun`

```
run(question):
  1. (multi-turn) classify_turn() -> new/refine/abandon; resolve_reference()
     - "abandon" short-circuits with a polite answer.
  2. ctx.cache["question"] = question
  3. system = _render_system(...)                 # built ONCE per run, re-sent every step
       - _select_tools(question): LLM intent classifier -> render only relevant tool group(s)
       - render_dataset_block(ctx) + render_tool_catalog(tools) + PLANNER_SYSTEM
  4. history = ["USER QUESTION: ..."]
  5. while n < max_steps:
       user_msg = "\n\n".join(history) + "You have {max_steps-n} tool calls left..."
       decision = parse(ctx.llm.complete(system, user_msg))
       if "final_answer": finalize + return
       tool = registry.get(decision["tool"])
       result = _call_tool(tool, decision["args"])   # pydantic-validate then tool.run()
       history.append("TOOL CALL ...\nOBSERVATION: " + result.summary[:1500])
       # anti-loop: same tool fails 2x in a row -> inject "[LOOP GUARD] re-plan" directive
       if result.meta.get("pause_for_user"): return (clarify pause)   # interactive only
  6. budget exhausted -> _force_final(): ask once for a final answer
```

Key methods: `run`, `_render_system`, `_select_tools`, `_call_tool`, `_force_final`.
Dispatch **always uses the full `self.registry`**; the tool selector only narrows
what the planner *sees* in the prompt, never what it can call.

Data classes:
- `AgentRun` — `run_id, question, dataset_id, steps[], final_answer, ok, error, pending_clarification, system_prompt`.
- `Step` — `n, thought, tool, args, result(ToolResult), raw_llm_text, latency_s, prompt`. The full per-turn user message lives on each Step (so trajectories are reproducible).

---

## 4. Core data structures ([context.py](../lexsi_ds/agent/context.py))

**`AgentContext`** — built once per run, passed to every tool:
- `dataset: DatasetHandle`, `run_id`
- `org / text_project / tab_project / tab_projects` — Lexsi SDK handles (may be None offline)
- `llm` — an `LLMClient`
- `cache: dict` — run-scoped scratchpad (see §6)
- `session` — optional `SessionContext` (multi-turn)
- `interactive: bool` — True in the UI (clarify pauses); False in bench/headless (clarify falls back to its `default`)
- `.duck(read_only=None)` — open a DuckDB connection on `dataset.duckdb_path` (read-only for bundled, writable otherwise)

**`DatasetHandle`** — `id, kind, duckdb_path, tables: list[TableInfo], connector_id, kg`. `kind ∈ {bundled, upload, connector, attached}`. Everything funnels through `ctx.duck()` → `duckdb_path`, so "switch dataset" = build a new handle and assign `ctx.dataset`.

**`TableInfo`** = `name, columns: list[ColumnInfo], n_rows, comment`. **`ColumnInfo`** = `name, dtype, comment, sample` (the `sample` value is the per-column representative value the planner sees in the schema — populated at describe time).

---

## 5. The tool contract — how to add a tool ([tools/base.py](../lexsi_ds/agent/tools/base.py))

A tool is any object with `.spec: ToolSpec` and `.run(args, ctx) -> ToolResult`.

```python
# lexsi_ds/agent/tools/my_tool.py
from pydantic import BaseModel, Field
from lexsi_ds.agent.context import AgentContext
from lexsi_ds.agent.tools.base import Tool, ToolResult, ToolSpec

class MyToolArgs(BaseModel):
    table_or_label: str = Field(..., description="...")   # match sibling arg names!

def _run(args: MyToolArgs, ctx: AgentContext) -> ToolResult:
    con = ctx.duck()
    ...
    return ToolResult(ok=True, summary="short text the LLM sees (<~200 tok)",
                      payload={"dataframe": df})           # rich data; LLM never sees it

class _MyTool:
    spec = ToolSpec(name="my_tool", description="...(LLM sees verbatim)...",
                    args_schema=MyToolArgs, returns="what the observation looks like")
    def run(self, args, ctx): return _run(args, ctx)

TOOL: Tool = _MyTool()
```

Then register it: add `from .my_tool import TOOL as my_tool_tool` and an entry in
`REGISTRY` in [tools/__init__.py](../lexsi_ds/agent/tools/__init__.py). If it fits an
intent group, add it to `_TOOL_GROUPS` in [prompts.py](../lexsi_ds/agent/prompts.py)
so the tool selector surfaces it (see §8).

Contract notes:
- `summary` is the **only** thing the LLM sees; `payload` is for the UI / downstream tools.
- Return `ok=False` with an **actionable** `summary` on expected failures — the planner reads it and re-plans. Unexpected exceptions propagate; the loop wraps them.
- **Arg-name convention:** sibling tools that take a source use `table_or_label`. Match it (or alias via pydantic `AliasChoices`) — mismatches show up as `validation_error` and waste planner steps (this bit us; see `transform_column`).
- Args are validated in `_call_tool` via `tool.spec.args_schema.model_validate(raw_args)`.

---

## 6. The cache contract (`ctx.cache`)

Run-scoped (session-scoped in multi-turn). Producers → consumers, by key:

| Key | Producer | Consumer |
|---|---|---|
| `question` | loop | tool selector |
| `tool_intents::<q>` | `_select_tools` | (memoize the intent classification) |
| `sql_result:<label>` | `run_sql` | profile_data, sample_rows, transform_column, predict |
| `last_sql_result`, `last_sql` | `run_sql` | PKDD scorer, downstream |
| `profile:<label>` | `profile_data` | train_tabular_model, dq checks |
| `last_train_columns` | `train_tabular_model` | predict (schema alignment) |
| `last_predictions`, `last_predict_tag`, `last_model_id` | `predict` | explain_prediction, evaluate_predictions |
| `last_eval` | `evaluate_predictions` | summarize_result |

When adding a tool that hands data downstream, write a documented key here.

---

## 7. Data layer ([datasource.py](../lexsi_ds/agent/datasource.py))

Handle loaders (each returns a `DatasetHandle`):
- `load_pkdd_handle()` — bundled PKDD (`DUCKDB_PATH = artifacts/financial.duckdb`), KG attached.
- `load_upload_handle(spec)` — CSV/Parquet → per-session DuckDB.
- `load_s3_connector_handle(spec)` — `read_parquet(s3://…)` views.
- `load_attached_handle(specs, *, dataset_id, target_path=None)` — fold heterogeneous DBs into **one** DuckDB.
- `load_postgres_connector_handle` / `load_mongo_connector_handle` — live DB connectors (reuse the materialize helpers).

**Multi-DB fold-in** (`_build_attached_duckdb`): each source becomes `<alias>_<table>`:
- sqlite/duckdb → `ATTACH` + `CREATE TABLE AS SELECT` (`_materialize_file_db`)
- postgres → restore the `.sql` dump into an embedded **pgserver**, copy via postgres scanner (`_materialize_postgres_dump`); tears down the `pgdata` dir after.
- mongo → read `.bson` server-lessly, flatten nested → JSON text (`_materialize_mongo_dump`).

**Schema sampling:** `_describe_tables` → `_column_samples` runs one `SELECT * LIMIT 5` per table and stores a representative value on `ColumnInfo.sample`. This is what lets the planner see "Date: VARCHAR sample='31 Dec 1986, 00:00'" and parse with `TRY_CAST`/`strptime` instead of naive casts.

**DB cache (build-once)** — `load_attached_handle(target_path=...)`: if the master exists at `target_path`, reuse it; else materialize once (atomic `.building` → rename). Each run gets an isolated **working copy** (`shutil.copyfile` → `artifacts/sessions/attach_*.duckdb`) so agent writes never mutate the master. The DAB adapter computes the master path + content-fingerprint (see §13).

Session files live in `artifacts/sessions/` via `_new_session_db()`; the DAB runner deletes the per-dataset working copy when done (`_cleanup_working_copy`).

---

## 8. Prompts + the token-lean tool selector ([prompts.py](../lexsi_ds/agent/prompts.py))

- `PLANNER_SYSTEM` — the ReAct framing + hard rules + JSON response format.
- `render_tool_catalog(registry)` — renders each tool as `## name` + description +
  **one-line arg signature** (`_compact_args`, not full JSON Schema — ~64% fewer tokens)
  + Returns. Enums/Literals keep their choices.
- `render_dataset_block(ctx)` / `render_session_context(...)` — dataset summary + multi-turn prefix.
- **Tool selector** (gated, LLM-based): `_select_tools` (in loop.py) calls a one-shot
  classifier (`TOOL_INTENT_SYSTEM`) once per run, cached. `parse_tool_intents` →
  `subset_for_intents(intents, registry)` renders **core tools + the union of matched
  groups** (`_CORE_TOOLS`, `_TOOL_GROUPS`); ambiguous/multi-intent → full catalog. Toggle
  with `LEXSI_TOOL_SELECTOR=0`. Dispatch is always against the full registry.

---

## 9. LLM client ([llm/client.py](../lexsi_ds/llm/client.py))

- `LLMClient` protocol: `.complete(system, user) -> LLMResult(text, raw)`.
- Implementations: `StubLLMClient` (offline sentinels), `EchoGoldLLMClient` (tests),
  `LexsiTextClient` (real; `LexsiTextClient.from_env()`), built via `factory("lexsi"|"stub")`.
- **Caching:** in-memory LRU (`LEXSI_LLM_CACHE_MAX`, default 256) keyed on
  `(model, provider, max_tokens, system, user)`; optional **disk** layer when
  `LEXSI_LLM_CACHE_DIR` is set (the bench runners set it → replays across runs). Disable
  with `LEXSI_LLM_CACHE=0`.
- **Timeout/retry (important):** every gateway call goes through `_call_with_timeout`
  (daemon-thread guard, `LEXSI_LLM_TIMEOUT_S` default 180, `LEXSI_LLM_RETRIES` default 1).
  Without it a stalled gateway request freezes the whole run — this was a real incident.
- **Model-family handling:** GPT-5 / o-series reject `max_tokens`; the client POSTs
  directly with `max_completion_tokens` (`_post_chat_with_completion_tokens`).

---

## 10. Multi-turn

`SessionContext` ([session.py](../lexsi_ds/agent/session.py)) seeds `ctx.cache` from the
session at run start and lifts changes back at the end. `classify_turn`
([turn_classifier.py](../lexsi_ds/agent/turn_classifier.py)) labels each turn
new/refine/abandon; `resolve` ([reference_resolver.py](../lexsi_ds/agent/reference_resolver.py))
resolves anaphora and hands the planner an explicit reference.

---

## 11. Configuration (environment variables)

**Required for the real LLM path:** `SDK_ACCESS_TOKEN`, `LEXSI_ORG_NAME`,
`LEXSI_WORKSPACE_NAME`, `LEXSI_TEXT_PROJECT_NAME`, `LEXSI_TEXT_PROVIDER`,
`LEXSI_TEXT_MODEL`. Optional: `LEXSI_TEXT_PROVIDER_API_KEY`, `LEXSI_TEXT_MAX_TOKENS`.

**Tabular:** `LEXSI_TABULAR_PROJECT_NAME` (per-run projects created on train, so usually unset).

| Var | Default | Purpose |
|---|---|---|
| `LEXSI_TOOL_SELECTOR` | `1` | gate the LLM tool selector (`0` = always full catalog) |
| `LEXSI_LLM_TIMEOUT_S` | `180` | hard timeout per gateway call |
| `LEXSI_LLM_RETRIES` | `1` | retries on timeout/failure |
| `LEXSI_LLM_CACHE` | `1` | in-memory LLM cache on/off |
| `LEXSI_LLM_CACHE_DIR` | — | enable on-disk LLM cache (bench runners set this) |
| `LEXSI_LLM_CACHE_MAX` | `256` | LRU size |
| `DUCKDB_PATH` | `artifacts/financial.duckdb` | bundled PKDD DB |
| `DAB_ROOT` | `benchmark/dab/DataAgentBench` | DAB checkout |
| `GRADIO_PORT` | `7860` (7850 in Docker) | UI port |
| `LEXSI_UI_RUN_TIMEOUT_S`, `LEXSI_UI_HEARTBEAT_S`, `LEXSI_UI_SSR` | — | UI tuning |
| `LEXSI_TRAIN_TIMEOUT_S`, `LEXSI_PREDICT_TIMEOUT_S`, `LEXSI_XAI_BUDGET_S` | — | Lexsi pod waits |
| `LEXSI_ROUNDTRIP_MODEL/_PROVIDER/_DISABLE` | — | datagen Stage-4 back-translator |

(Connector creds — `PG_*`, `MONGO_URI`, `AWS_*`, `MYSQL_*` — are read by the live `connect_datalake` paths.)

---

## 12. Local dev setup

```bash
# deps (extras: lexsi, ui, dev, embeddings, connectors)
uv sync --extra lexsi --extra ui --extra dev --extra embeddings --extra connectors

# bootstrap the bundled PKDD DuckDB (one-time, ~10s)
uv run python -m lexsi_ds.data.loader

# drive the agent from Python (offline stub LLM)
uv run python -c "
from lexsi_ds.agent import AgentContext, AgentLoop
from lexsi_ds.agent.datasource import load_pkdd_handle
from lexsi_ds.llm.client import factory
ctx = AgentContext(dataset=load_pkdd_handle(), run_id='local', llm=factory('stub'))
print(AgentLoop(ctx=ctx).run('Top 5 districts by number of accounts.').final_answer)
"

# UI
uv run python -m app.gradio_app          # http://localhost:7860

# tests (LLM-gated ones skip without the Lexsi env)
uv run pytest -q
```

---

## 13. Benchmark mechanisms (all of them)

There are **three** harnesses. All write full trajectories so failures are debuggable.

### 13a. DataAgentBench (external; Pass@1) — `benchmark/dab/`

What it is: 17 datasets across sqlite/duckdb/postgres/mongo; each task ships a
question + `ground_truth.csv` + its own `validate.py`. We score **Pass@1** by calling
that `validate.py` on the agent's final answer.

Code:
- [adapter.py](../benchmark/dab/adapter.py): `load_dataset(dir)` → `DabDataset`
  (parses `db_config.yaml` into `AttachSpec`s via `parse_db_config`, loads each task's
  `validate.py` via `_load_validate`). `build_handle(dataset, cache_dir=...)` →
  `load_attached_handle` with a content-keyed master path
  (`<cache_dir>/<dataset>__<_fingerprint>.duckdb`).
- [runner.py](../benchmark/dab/runner.py): `main` → `run_dataset` (builds the handle
  once per dataset, loops tasks) → `run_task` (composes the question with the DAB
  description + table-name mapping, runs `AgentLoop`, calls `validate`). DB working
  copy cleaned per dataset (`_cleanup_working_copy`).

Setup + run:
```bash
git clone https://github.com/ucbepic/DataAgentBench benchmark/dab/DataAgentBench
uv sync --extra connectors        # needed for the postgres/mongo datasets
export SDK_ACCESS_TOKEN=… LEXSI_WORKSPACE_NAME=… LEXSI_TEXT_PROJECT_NAME=…  # + org/model/provider

# one dataset / one task / sweep all
uv run python -m benchmark.dab.runner --dataset DEPS_DEV_V1 --llm lexsi
uv run python -m benchmark.dab.runner --dataset DEPS_DEV_V1 --query query1 --llm lexsi
uv run python -m benchmark.dab.runner --all --llm lexsi

# the sqlite/duckdb "core" (5 datasets / 17 tasks, no connectors extra), real Pass@1
uv run python -m benchmark.dab.runner --all --db-types sqlite,duckdb --llm lexsi \
  --iterations 3 --max-steps 20 --log-dir artifacts/dab_runs/run1
```

Flags: `--dataset/--all`, `--db-types t1,t2` (filter `--all`), `--query`,
`--iterations`, `--max-steps` (default 12), `--use-hints`, `--log-dir`,
`--db-cache-dir`/`--no-db-cache`, `--llm-cache-dir`/`--no-llm-cache`, `-v`.

Caching (two layers, both speed re-runs):
- **DB cache** `artifacts/dab_dbs/` — each dataset's fold-in materialized once; reused across runs; per-run working copy auto-deleted.
- **LLM disk cache** `artifacts/llm_cache/` — gateway responses keyed on the prompt; a failed sweep resumes cheaply.

Outputs: per-task trace JSON in `--log-dir` (`<dataset>_<query>_iter<N>.json` — full
steps, prompts, raw LLM output, payloads, validator verdict) + a console scoreboard
(per-dataset pass counts, wall vs agent time, LLM reuse). Latest results writeup:
[`dab_results_v1.md`](dab_results_v1.md); integration detail: [`dab_integration.md`](dab_integration.md).

### 13b. PKDD-Curated (internal) — `benchmark/pkdd_runner.py`

What it is: our own suite over the bundled PKDD dataset, scored against the **pinned
fixture** `benchmark/fixtures/pkdd_seed.duckdb`. Question schema in
[`benchmark/datagen/types.py`](../benchmark/datagen/types.py) (`Row`); the live set is
[`bench/pkdd/questions.yaml`](../bench/pkdd/questions.yaml) (40 items: analytic,
predictive, adversarial).

Scoring ([pkdd_runner.py](../benchmark/pkdd_runner.py), by `scoring_rule`):
- `row_set_equality` (analytic) → `_score_row_set`: agent's `last_sql_result` vs
  `gold_sql` executed on the fixture, as a multiset of sorted-value rows (column-rename
  tolerant; lenient scalar fallback).
- `exact_match` (numeric) → `_score_numeric`: reference value within `tolerance`.
- `judge_rubric` (adversarial) → `_score_behavioral`: heuristic — fail if it used a
  `forbidden_tool` or answered when it should clarify/refuse.
- `ndcg@k` (predictive) → captured but **not auto-scored** unless `--include-predictive`
  (those train real models, slow). Plan precision/recall computed for every row.

Run:
```bash
export SDK_ACCESS_TOKEN=… LEXSI_WORKSPACE_NAME=… LEXSI_TEXT_PROJECT_NAME=…  # + org/model/provider
uv run python -m benchmark.pkdd_runner --llm lexsi --log-dir artifacts/pkdd_runs
# scope / debug
uv run python -m benchmark.pkdd_runner --tier analytic --limit 5 --llm lexsi
uv run python -m benchmark.pkdd_runner --include-predictive --llm lexsi
```
Flags: `--tier`, `--include-predictive`, `--limit`, `--llm`, `--log-dir`,
`--llm-cache-dir`/`--no-llm-cache`, `-v`. Outputs: per-row trace JSON + a per-tier
scoreboard (pass/scored + plan p/r). The agent queries the fixture (handle from
`load_pkdd_handle()` with `duckdb_path` repointed to the fixture).

> Note: `bench/pkdd/sessions.yaml` (multi-turn) and `p2_feature_engineering.yaml`
> (separate `assertions` schema) are **not** run by `pkdd_runner` yet.

### 13c. Datagen pipeline (generates the PKDD questions) — `benchmark/datagen/`

Template-based; authoring is per-template, so ~10 templates yield 500+ rows. Stages:
column inventory → templates → instantiate → execute gold SQL on the pinned fixture →
LLM round-trip verify → predictive enrichment → multi-turn compose → emit YAML.

```bash
uv run python -m benchmark.datagen pin-fixture        # snapshot artifacts/financial.duckdb
uv run python -m benchmark.datagen run --skip-roundtrip   # fast (skip LLM Stage 4)
uv run python -m benchmark.datagen run                    # full (needs Lexsi text env)
```
Every emitted row carries `fixture_sha`; a scoreboard from a different sha isn't comparable.
The grading design (5 axes, tiers, pass^k) lives in [`v1_benchmarking.md`](v1_benchmarking.md).

### 13d. SDK repro / probe scripts — `scripts/repro_*`

Standalone, self-contained scripts that hit the live Lexsi SDK against a real
trained model and report OK/FAIL per call. They're how we validate the
SDK-backed tools and file platform bugs. Each catches every call so one run
produces a complete report; each leaves its project on the platform for
Activity-Log inspection.

- **`scripts/repro_tabicl/`** — reproduces TabICL `model_inference` + per-case XAI
  failures (empty `Exception` on T4 pods, inference hang/timeout, `case_predict`
  `ValidationError`). Ships its own PKDD CSVs in `data/`.
- **`scripts/repro_model_lifecycle/`** — probe harness for the **model-lifecycle
  tools** (`list_available_models`, `select_active_model`, `compare_models`,
  `check_drift`, `monitor_performance`) + the future explanation methods. Trains
  TabICL + XGBoost on the (local) PKDD CSVs and calls each SDK method. Two scripts:
  `repro_model_lifecycle.py` (TabICL-first, the clean path) and
  `repro_model_lifecycle_xgboost_first.py` (XGBoost-first, reproduces the `#05-013`
  cascade). Run: `uv run python scripts/repro_model_lifecycle/repro_model_lifecycle.py`
  (env: `SDK_ACCESS_TOKEN`, `LEXSI_WORKSPACE_NAME`, `LEXSI_ORG_NAME`).

**Live SDK findings from the lifecycle repro (last run 13 OK / 4 FAIL):**
- `#05-013` — classic-ML (XGBoost) explainability scans **every stored column,
  including the server-injected `tag` column**, and fails `float()` on a
  non-numeric one. The SDK uploads clean data + an explicit `feature_include`
  (`tabular.py:285,368`); explainability ignores it. Model still **builds**
  (non-fatal), but `upload_data` raises and **rolls back the upload** → cascade
  (`train_model` → "Upload files first", predict upload → "Project Config is
  required"). TabICL's foundation path doesn't trip it. **Fix for the agent:**
  treat `#05-013` as success-with-warning, and train foundation-first / drop or
  encode non-numeric columns.
- `evals_tabular(<foundation model>)` → `Exception: 'model'` (KeyError) — works
  for classic ML, breaks for TabICL. Blocks `compare_models` on foundation models.
- `get_model_performance(...)` → `'utf-8' codec can't decode …` — response decode
  bug. Blocks `monitor_performance`.
- per-case XAI (`case_predict`, `get_feature_importance`) needs (a) the model
  **active** + an inference run first (else "Inference status inactive"), and
  (b) SHAP **computed at train** (`xai=["shap"]`) — which itself trips `#05-013`
  for classic ML. `case_predict(risk_policies=True)` works without SHAP.

The five lifecycle tools degrade gracefully against these (per-model error capture
in `compare_models`; actionable `ok=False` in `monitor_performance`). Record new
findings in [`sdk_issues.md`](sdk_issues.md).

---

## 14. Deploy

- **Docker** ([Dockerfile](../Dockerfile)): `python:3.10-slim`, CPU-only torch, pinned
  deps, **bakes** `artifacts/financial.duckdb` (the PKDD source TSVs are Git-LFS
  pointers), serves Gradio on **7850**. Build for the platform arch:
  `docker buildx build --platform linux/amd64 --provenance=false --sbom=false -t bplexsi/lexsi-ds-agent:prod_vN --push .`
- **HF Space** — a Gradio Space (`README.md` frontmatter `sdk: gradio`,
  `app_file: app/gradio_app.py`); the Dockerfile is ignored there.
- **Lexsi platform** — [config.yaml](../config.yaml): one container on port 7850; the
  real `SDK_ACCESS_TOKEN` goes in the platform env tab (the repo keeps a placeholder).

---

## 15. Sharp edges (things that bit us)

- **No LLM timeout = frozen run.** Always keep `_call_with_timeout` in the path (§9).
- **Tool arg-name drift.** The planner reuses sibling names (`table_or_label`,
  `source_columns`, `prompt`); mismatched schemas → `validation_error` loops. Alias them.
- **DataFrame truthiness.** `x = cache.get(a) or cache.get(b)` raises on a DataFrame; use explicit `is None`.
- **String-typed dates/numbers.** Source columns are often VARCHAR; rely on the
  `ColumnInfo.sample` + the `TRY_CAST`/`try_strptime` guidance in the text_to_sql prompt.
- **DB cache contamination.** The master must stay pristine — agents write to the
  working copy, never the master (§7).
- **`artifacts/sessions/` growth.** Working copies/pgdata are cleaned now; if you add a
  loader, clean up after it.
- **Single-run bench numbers are noisy** (DAB sits in an 8–9/17 band, 11/17 ceiling).
  Use `--iterations` and read the per-task/failure breakdown, not the headline.

---

## 16. "Where do I look for…"

| Want to… | Go to |
|---|---|
| change the planner behavior / rules | `prompts.py` (`PLANNER_SYSTEM`) |
| add/modify a tool | `tools/<name>.py` + `tools/__init__.py` (+ `_TOOL_GROUPS`) |
| change which tools the planner sees | `prompts.py` selector (`_CORE_TOOLS`, `_TOOL_GROUPS`) |
| connect a new datasource | `datasource.py` (loaders) + `connect_datalake` tool |
| change LLM / caching / timeout | `llm/client.py` |
| run/extend DAB | `benchmark/dab/{adapter,runner}.py` |
| run/extend PKDD | `benchmark/pkdd_runner.py` + `bench/pkdd/questions.yaml` |
| regenerate PKDD questions | `benchmark/datagen/` |
| model-lifecycle tools (list/select/compare/drift/monitor) | `tools/{list_available_models,select_active_model,compare_models,check_drift,monitor_performance}.py` + `tools/_tabular_common.py` |
| probe the live SDK / file an SDK bug | `scripts/repro_model_lifecycle/`, `scripts/repro_tabicl/` → `docs/sdk_issues.md` |
| the UI | `app/gradio_app.py` |
| roadmap / what's next | `v1_tools.md`, `tools_remaining_and_dab_hardening.md` |