Spaces:
Sleeping
Sleeping
| # Lexsi DS Agent — Developer / KT Guide | |
| A code-level companion to [`DSAGENT_OVERVIEW.md`](DSAGENT_OVERVIEW.md) (which is | |
| the what/why). This is the **how**: file map, control flow, the contracts you'll | |
| touch, configuration, and the full benchmark-running mechanics. Read the overview | |
| first, then this. | |
| --- | |
| ## 1. Mental model in one paragraph | |
| One **question** → one `AgentLoop.run()` → a **ReAct loop**: build a system prompt | |
| (framing + active-dataset summary + tool catalog), ask the LLM for a JSON decision | |
| (`{"thought","tool","args"}` or `{"thought","final_answer"}`), validate args with | |
| the tool's pydantic schema, run the tool against a shared `AgentContext`, append the | |
| observation to history, repeat until a final answer or the step budget. Tools read | |
| and write a shared `ctx.cache` scratchpad. The data substrate is always **DuckDB**; | |
| tabular ML runs on **remote Lexsi pods** via the SDK. The loop is generic — it has | |
| no opinion on which tools to call. | |
| --- | |
| ## 2. Repo layout (code-oriented) | |
| ``` | |
| lexsi_ds/ | |
| agent/ | |
| loop.py AgentLoop.run() — the ReAct orchestrator (start here) | |
| context.py AgentContext, DatasetHandle, TableInfo, ColumnInfo | |
| prompts.py PLANNER_SYSTEM, render_tool_catalog, tool-selector (intent routing) | |
| datasource.py DuckDB handle loaders + multi-DB materialization + DB cache | |
| tools/ | |
| base.py Tool / ToolSpec / ToolResult contract (the interface) | |
| __init__.py REGISTRY: dict[str, Tool] (30 tools) | |
| <name>.py one tool per file, each exports `TOOL: Tool` | |
| _tabular_common.py shared helpers for the model-lifecycle tools | |
| (resolve_tab_project, active_model_name, df_records, …) | |
| kg_index.py embedding/lexical index over the PKDD knowledge graph | |
| table_index.py embedding index over table schemas (schema linking, big DBs) | |
| session.py SessionContext + multi-turn merge | |
| turn_classifier.py classify a turn: new / refine / abandon | |
| reference_resolver.py resolve "it/that/those" against prior turns | |
| samples.py curated demo questions (by_id, SAMPLES) — UI chips + tests | |
| llm/client.py LLMClient protocol, StubLLMClient, LexsiTextClient, cache, timeout | |
| schema/ | |
| raw.py PKDD DDL introspection (introspect()) | |
| context_graph.py ContextGraph — the banking knowledge graph | |
| data/loader.py PKDD TSVs -> artifacts/financial.duckdb | |
| paths.py DUCKDB_PATH, ARTIFACTS_DIR, etc. | |
| app/gradio_app.py the UI (build_ui(); demo.launch on GRADIO_PORT) | |
| benchmark/ | |
| dab/ | |
| adapter.py parse a DAB dataset -> DatasetHandle + tasks (+ DB cache key) | |
| runner.py run agent per DAB task, score Pass@1 via the task's validate.py | |
| DataAgentBench/ the (gitignored) upstream checkout + DB files | |
| pkdd_runner.py run agent over bench/pkdd/questions.yaml, auto-score | |
| datagen/ template pipeline that GENERATES bench/pkdd/*.yaml | |
| bench/pkdd/ | |
| questions.yaml 40 single-turn bench items (analytic/predictive/adversarial) | |
| sessions.yaml multi-turn sessions | |
| p2_feature_engineering.yaml 10 feature-eng items (separate `assertions` schema) | |
| benchmark/fixtures/pkdd_seed.duckdb pinned PKDD snapshot the bench scores against | |
| scripts/ | |
| repro_tabicl/ TabICL inference + per-case XAI failure repro | |
| repro_model_lifecycle/ model-lifecycle SDK probe harness (self-contained): | |
| data/ the PKDD CSVs (local copy) | |
| repro_model_lifecycle.py TabICL-first (clean) | |
| repro_model_lifecycle_xgboost_first.py XGBoost-first (#05-013 cascade repro) | |
| tests/ pytest suite (test_b_tools, test_v1_multiturn, test_dab_*, …) | |
| docs/ design + results + this guide | |
| ``` | |
| --- | |
| ## 3. The run lifecycle ([loop.py](../lexsi_ds/agent/loop.py)) | |
| `AgentLoop(ctx, registry=REGISTRY, max_steps=18).run(question) -> AgentRun` | |
| ``` | |
| run(question): | |
| 1. (multi-turn) classify_turn() -> new/refine/abandon; resolve_reference() | |
| - "abandon" short-circuits with a polite answer. | |
| 2. ctx.cache["question"] = question | |
| 3. system = _render_system(...) # built ONCE per run, re-sent every step | |
| - _select_tools(question): LLM intent classifier -> render only relevant tool group(s) | |
| - render_dataset_block(ctx) + render_tool_catalog(tools) + PLANNER_SYSTEM | |
| 4. history = ["USER QUESTION: ..."] | |
| 5. while n < max_steps: | |
| user_msg = "\n\n".join(history) + "You have {max_steps-n} tool calls left..." | |
| decision = parse(ctx.llm.complete(system, user_msg)) | |
| if "final_answer": finalize + return | |
| tool = registry.get(decision["tool"]) | |
| result = _call_tool(tool, decision["args"]) # pydantic-validate then tool.run() | |
| history.append("TOOL CALL ...\nOBSERVATION: " + result.summary[:1500]) | |
| # anti-loop: same tool fails 2x in a row -> inject "[LOOP GUARD] re-plan" directive | |
| if result.meta.get("pause_for_user"): return (clarify pause) # interactive only | |
| 6. budget exhausted -> _force_final(): ask once for a final answer | |
| ``` | |
| Key methods: `run`, `_render_system`, `_select_tools`, `_call_tool`, `_force_final`. | |
| Dispatch **always uses the full `self.registry`**; the tool selector only narrows | |
| what the planner *sees* in the prompt, never what it can call. | |
| Data classes: | |
| - `AgentRun` — `run_id, question, dataset_id, steps[], final_answer, ok, error, pending_clarification, system_prompt`. | |
| - `Step` — `n, thought, tool, args, result(ToolResult), raw_llm_text, latency_s, prompt`. The full per-turn user message lives on each Step (so trajectories are reproducible). | |
| --- | |
| ## 4. Core data structures ([context.py](../lexsi_ds/agent/context.py)) | |
| **`AgentContext`** — built once per run, passed to every tool: | |
| - `dataset: DatasetHandle`, `run_id` | |
| - `org / text_project / tab_project / tab_projects` — Lexsi SDK handles (may be None offline) | |
| - `llm` — an `LLMClient` | |
| - `cache: dict` — run-scoped scratchpad (see §6) | |
| - `session` — optional `SessionContext` (multi-turn) | |
| - `interactive: bool` — True in the UI (clarify pauses); False in bench/headless (clarify falls back to its `default`) | |
| - `.duck(read_only=None)` — open a DuckDB connection on `dataset.duckdb_path` (read-only for bundled, writable otherwise) | |
| **`DatasetHandle`** — `id, kind, duckdb_path, tables: list[TableInfo], connector_id, kg`. `kind ∈ {bundled, upload, connector, attached}`. Everything funnels through `ctx.duck()` → `duckdb_path`, so "switch dataset" = build a new handle and assign `ctx.dataset`. | |
| **`TableInfo`** = `name, columns: list[ColumnInfo], n_rows, comment`. **`ColumnInfo`** = `name, dtype, comment, sample` (the `sample` value is the per-column representative value the planner sees in the schema — populated at describe time). | |
| --- | |
| ## 5. The tool contract — how to add a tool ([tools/base.py](../lexsi_ds/agent/tools/base.py)) | |
| A tool is any object with `.spec: ToolSpec` and `.run(args, ctx) -> ToolResult`. | |
| ```python | |
| # lexsi_ds/agent/tools/my_tool.py | |
| from pydantic import BaseModel, Field | |
| from lexsi_ds.agent.context import AgentContext | |
| from lexsi_ds.agent.tools.base import Tool, ToolResult, ToolSpec | |
| class MyToolArgs(BaseModel): | |
| table_or_label: str = Field(..., description="...") # match sibling arg names! | |
| def _run(args: MyToolArgs, ctx: AgentContext) -> ToolResult: | |
| con = ctx.duck() | |
| ... | |
| return ToolResult(ok=True, summary="short text the LLM sees (<~200 tok)", | |
| payload={"dataframe": df}) # rich data; LLM never sees it | |
| class _MyTool: | |
| spec = ToolSpec(name="my_tool", description="...(LLM sees verbatim)...", | |
| args_schema=MyToolArgs, returns="what the observation looks like") | |
| def run(self, args, ctx): return _run(args, ctx) | |
| TOOL: Tool = _MyTool() | |
| ``` | |
| Then register it: add `from .my_tool import TOOL as my_tool_tool` and an entry in | |
| `REGISTRY` in [tools/__init__.py](../lexsi_ds/agent/tools/__init__.py). If it fits an | |
| intent group, add it to `_TOOL_GROUPS` in [prompts.py](../lexsi_ds/agent/prompts.py) | |
| so the tool selector surfaces it (see §8). | |
| Contract notes: | |
| - `summary` is the **only** thing the LLM sees; `payload` is for the UI / downstream tools. | |
| - Return `ok=False` with an **actionable** `summary` on expected failures — the planner reads it and re-plans. Unexpected exceptions propagate; the loop wraps them. | |
| - **Arg-name convention:** sibling tools that take a source use `table_or_label`. Match it (or alias via pydantic `AliasChoices`) — mismatches show up as `validation_error` and waste planner steps (this bit us; see `transform_column`). | |
| - Args are validated in `_call_tool` via `tool.spec.args_schema.model_validate(raw_args)`. | |
| --- | |
| ## 6. The cache contract (`ctx.cache`) | |
| Run-scoped (session-scoped in multi-turn). Producers → consumers, by key: | |
| | Key | Producer | Consumer | | |
| |---|---|---| | |
| | `question` | loop | tool selector | | |
| | `tool_intents::<q>` | `_select_tools` | (memoize the intent classification) | | |
| | `sql_result:<label>` | `run_sql` | profile_data, sample_rows, transform_column, predict | | |
| | `last_sql_result`, `last_sql` | `run_sql` | PKDD scorer, downstream | | |
| | `profile:<label>` | `profile_data` | train_tabular_model, dq checks | | |
| | `last_train_columns` | `train_tabular_model` | predict (schema alignment) | | |
| | `last_predictions`, `last_predict_tag`, `last_model_id` | `predict` | explain_prediction, evaluate_predictions | | |
| | `last_eval` | `evaluate_predictions` | summarize_result | | |
| When adding a tool that hands data downstream, write a documented key here. | |
| --- | |
| ## 7. Data layer ([datasource.py](../lexsi_ds/agent/datasource.py)) | |
| Handle loaders (each returns a `DatasetHandle`): | |
| - `load_pkdd_handle()` — bundled PKDD (`DUCKDB_PATH = artifacts/financial.duckdb`), KG attached. | |
| - `load_upload_handle(spec)` — CSV/Parquet → per-session DuckDB. | |
| - `load_s3_connector_handle(spec)` — `read_parquet(s3://…)` views. | |
| - `load_attached_handle(specs, *, dataset_id, target_path=None)` — fold heterogeneous DBs into **one** DuckDB. | |
| - `load_postgres_connector_handle` / `load_mongo_connector_handle` — live DB connectors (reuse the materialize helpers). | |
| **Multi-DB fold-in** (`_build_attached_duckdb`): each source becomes `<alias>_<table>`: | |
| - sqlite/duckdb → `ATTACH` + `CREATE TABLE AS SELECT` (`_materialize_file_db`) | |
| - postgres → restore the `.sql` dump into an embedded **pgserver**, copy via postgres scanner (`_materialize_postgres_dump`); tears down the `pgdata` dir after. | |
| - mongo → read `.bson` server-lessly, flatten nested → JSON text (`_materialize_mongo_dump`). | |
| **Schema sampling:** `_describe_tables` → `_column_samples` runs one `SELECT * LIMIT 5` per table and stores a representative value on `ColumnInfo.sample`. This is what lets the planner see "Date: VARCHAR sample='31 Dec 1986, 00:00'" and parse with `TRY_CAST`/`strptime` instead of naive casts. | |
| **DB cache (build-once)** — `load_attached_handle(target_path=...)`: if the master exists at `target_path`, reuse it; else materialize once (atomic `.building` → rename). Each run gets an isolated **working copy** (`shutil.copyfile` → `artifacts/sessions/attach_*.duckdb`) so agent writes never mutate the master. The DAB adapter computes the master path + content-fingerprint (see §13). | |
| Session files live in `artifacts/sessions/` via `_new_session_db()`; the DAB runner deletes the per-dataset working copy when done (`_cleanup_working_copy`). | |
| --- | |
| ## 8. Prompts + the token-lean tool selector ([prompts.py](../lexsi_ds/agent/prompts.py)) | |
| - `PLANNER_SYSTEM` — the ReAct framing + hard rules + JSON response format. | |
| - `render_tool_catalog(registry)` — renders each tool as `## name` + description + | |
| **one-line arg signature** (`_compact_args`, not full JSON Schema — ~64% fewer tokens) | |
| + Returns. Enums/Literals keep their choices. | |
| - `render_dataset_block(ctx)` / `render_session_context(...)` — dataset summary + multi-turn prefix. | |
| - **Tool selector** (gated, LLM-based): `_select_tools` (in loop.py) calls a one-shot | |
| classifier (`TOOL_INTENT_SYSTEM`) once per run, cached. `parse_tool_intents` → | |
| `subset_for_intents(intents, registry)` renders **core tools + the union of matched | |
| groups** (`_CORE_TOOLS`, `_TOOL_GROUPS`); ambiguous/multi-intent → full catalog. Toggle | |
| with `LEXSI_TOOL_SELECTOR=0`. Dispatch is always against the full registry. | |
| --- | |
| ## 9. LLM client ([llm/client.py](../lexsi_ds/llm/client.py)) | |
| - `LLMClient` protocol: `.complete(system, user) -> LLMResult(text, raw)`. | |
| - Implementations: `StubLLMClient` (offline sentinels), `EchoGoldLLMClient` (tests), | |
| `LexsiTextClient` (real; `LexsiTextClient.from_env()`), built via `factory("lexsi"|"stub")`. | |
| - **Caching:** in-memory LRU (`LEXSI_LLM_CACHE_MAX`, default 256) keyed on | |
| `(model, provider, max_tokens, system, user)`; optional **disk** layer when | |
| `LEXSI_LLM_CACHE_DIR` is set (the bench runners set it → replays across runs). Disable | |
| with `LEXSI_LLM_CACHE=0`. | |
| - **Timeout/retry (important):** every gateway call goes through `_call_with_timeout` | |
| (daemon-thread guard, `LEXSI_LLM_TIMEOUT_S` default 180, `LEXSI_LLM_RETRIES` default 1). | |
| Without it a stalled gateway request freezes the whole run — this was a real incident. | |
| - **Model-family handling:** GPT-5 / o-series reject `max_tokens`; the client POSTs | |
| directly with `max_completion_tokens` (`_post_chat_with_completion_tokens`). | |
| --- | |
| ## 10. Multi-turn | |
| `SessionContext` ([session.py](../lexsi_ds/agent/session.py)) seeds `ctx.cache` from the | |
| session at run start and lifts changes back at the end. `classify_turn` | |
| ([turn_classifier.py](../lexsi_ds/agent/turn_classifier.py)) labels each turn | |
| new/refine/abandon; `resolve` ([reference_resolver.py](../lexsi_ds/agent/reference_resolver.py)) | |
| resolves anaphora and hands the planner an explicit reference. | |
| --- | |
| ## 11. Configuration (environment variables) | |
| **Required for the real LLM path:** `SDK_ACCESS_TOKEN`, `LEXSI_ORG_NAME`, | |
| `LEXSI_WORKSPACE_NAME`, `LEXSI_TEXT_PROJECT_NAME`, `LEXSI_TEXT_PROVIDER`, | |
| `LEXSI_TEXT_MODEL`. Optional: `LEXSI_TEXT_PROVIDER_API_KEY`, `LEXSI_TEXT_MAX_TOKENS`. | |
| **Tabular:** `LEXSI_TABULAR_PROJECT_NAME` (per-run projects created on train, so usually unset). | |
| | Var | Default | Purpose | | |
| |---|---|---| | |
| | `LEXSI_TOOL_SELECTOR` | `1` | gate the LLM tool selector (`0` = always full catalog) | | |
| | `LEXSI_LLM_TIMEOUT_S` | `180` | hard timeout per gateway call | | |
| | `LEXSI_LLM_RETRIES` | `1` | retries on timeout/failure | | |
| | `LEXSI_LLM_CACHE` | `1` | in-memory LLM cache on/off | | |
| | `LEXSI_LLM_CACHE_DIR` | — | enable on-disk LLM cache (bench runners set this) | | |
| | `LEXSI_LLM_CACHE_MAX` | `256` | LRU size | | |
| | `DUCKDB_PATH` | `artifacts/financial.duckdb` | bundled PKDD DB | | |
| | `DAB_ROOT` | `benchmark/dab/DataAgentBench` | DAB checkout | | |
| | `GRADIO_PORT` | `7860` (7850 in Docker) | UI port | | |
| | `LEXSI_UI_RUN_TIMEOUT_S`, `LEXSI_UI_HEARTBEAT_S`, `LEXSI_UI_SSR` | — | UI tuning | | |
| | `LEXSI_TRAIN_TIMEOUT_S`, `LEXSI_PREDICT_TIMEOUT_S`, `LEXSI_XAI_BUDGET_S` | — | Lexsi pod waits | | |
| | `LEXSI_ROUNDTRIP_MODEL/_PROVIDER/_DISABLE` | — | datagen Stage-4 back-translator | | |
| (Connector creds — `PG_*`, `MONGO_URI`, `AWS_*`, `MYSQL_*` — are read by the live `connect_datalake` paths.) | |
| --- | |
| ## 12. Local dev setup | |
| ```bash | |
| # deps (extras: lexsi, ui, dev, embeddings, connectors) | |
| uv sync --extra lexsi --extra ui --extra dev --extra embeddings --extra connectors | |
| # bootstrap the bundled PKDD DuckDB (one-time, ~10s) | |
| uv run python -m lexsi_ds.data.loader | |
| # drive the agent from Python (offline stub LLM) | |
| uv run python -c " | |
| from lexsi_ds.agent import AgentContext, AgentLoop | |
| from lexsi_ds.agent.datasource import load_pkdd_handle | |
| from lexsi_ds.llm.client import factory | |
| ctx = AgentContext(dataset=load_pkdd_handle(), run_id='local', llm=factory('stub')) | |
| print(AgentLoop(ctx=ctx).run('Top 5 districts by number of accounts.').final_answer) | |
| " | |
| # UI | |
| uv run python -m app.gradio_app # http://localhost:7860 | |
| # tests (LLM-gated ones skip without the Lexsi env) | |
| uv run pytest -q | |
| ``` | |
| --- | |
| ## 13. Benchmark mechanisms (all of them) | |
| There are **three** harnesses. All write full trajectories so failures are debuggable. | |
| ### 13a. DataAgentBench (external; Pass@1) — `benchmark/dab/` | |
| What it is: 17 datasets across sqlite/duckdb/postgres/mongo; each task ships a | |
| question + `ground_truth.csv` + its own `validate.py`. We score **Pass@1** by calling | |
| that `validate.py` on the agent's final answer. | |
| Code: | |
| - [adapter.py](../benchmark/dab/adapter.py): `load_dataset(dir)` → `DabDataset` | |
| (parses `db_config.yaml` into `AttachSpec`s via `parse_db_config`, loads each task's | |
| `validate.py` via `_load_validate`). `build_handle(dataset, cache_dir=...)` → | |
| `load_attached_handle` with a content-keyed master path | |
| (`<cache_dir>/<dataset>__<_fingerprint>.duckdb`). | |
| - [runner.py](../benchmark/dab/runner.py): `main` → `run_dataset` (builds the handle | |
| once per dataset, loops tasks) → `run_task` (composes the question with the DAB | |
| description + table-name mapping, runs `AgentLoop`, calls `validate`). DB working | |
| copy cleaned per dataset (`_cleanup_working_copy`). | |
| Setup + run: | |
| ```bash | |
| git clone https://github.com/ucbepic/DataAgentBench benchmark/dab/DataAgentBench | |
| uv sync --extra connectors # needed for the postgres/mongo datasets | |
| export SDK_ACCESS_TOKEN=… LEXSI_WORKSPACE_NAME=… LEXSI_TEXT_PROJECT_NAME=… # + org/model/provider | |
| # one dataset / one task / sweep all | |
| uv run python -m benchmark.dab.runner --dataset DEPS_DEV_V1 --llm lexsi | |
| uv run python -m benchmark.dab.runner --dataset DEPS_DEV_V1 --query query1 --llm lexsi | |
| uv run python -m benchmark.dab.runner --all --llm lexsi | |
| # the sqlite/duckdb "core" (5 datasets / 17 tasks, no connectors extra), real Pass@1 | |
| uv run python -m benchmark.dab.runner --all --db-types sqlite,duckdb --llm lexsi \ | |
| --iterations 3 --max-steps 20 --log-dir artifacts/dab_runs/run1 | |
| ``` | |
| Flags: `--dataset/--all`, `--db-types t1,t2` (filter `--all`), `--query`, | |
| `--iterations`, `--max-steps` (default 12), `--use-hints`, `--log-dir`, | |
| `--db-cache-dir`/`--no-db-cache`, `--llm-cache-dir`/`--no-llm-cache`, `-v`. | |
| Caching (two layers, both speed re-runs): | |
| - **DB cache** `artifacts/dab_dbs/` — each dataset's fold-in materialized once; reused across runs; per-run working copy auto-deleted. | |
| - **LLM disk cache** `artifacts/llm_cache/` — gateway responses keyed on the prompt; a failed sweep resumes cheaply. | |
| Outputs: per-task trace JSON in `--log-dir` (`<dataset>_<query>_iter<N>.json` — full | |
| steps, prompts, raw LLM output, payloads, validator verdict) + a console scoreboard | |
| (per-dataset pass counts, wall vs agent time, LLM reuse). Latest results writeup: | |
| [`dab_results_v1.md`](dab_results_v1.md); integration detail: [`dab_integration.md`](dab_integration.md). | |
| ### 13b. PKDD-Curated (internal) — `benchmark/pkdd_runner.py` | |
| What it is: our own suite over the bundled PKDD dataset, scored against the **pinned | |
| fixture** `benchmark/fixtures/pkdd_seed.duckdb`. Question schema in | |
| [`benchmark/datagen/types.py`](../benchmark/datagen/types.py) (`Row`); the live set is | |
| [`bench/pkdd/questions.yaml`](../bench/pkdd/questions.yaml) (40 items: analytic, | |
| predictive, adversarial). | |
| Scoring ([pkdd_runner.py](../benchmark/pkdd_runner.py), by `scoring_rule`): | |
| - `row_set_equality` (analytic) → `_score_row_set`: agent's `last_sql_result` vs | |
| `gold_sql` executed on the fixture, as a multiset of sorted-value rows (column-rename | |
| tolerant; lenient scalar fallback). | |
| - `exact_match` (numeric) → `_score_numeric`: reference value within `tolerance`. | |
| - `judge_rubric` (adversarial) → `_score_behavioral`: heuristic — fail if it used a | |
| `forbidden_tool` or answered when it should clarify/refuse. | |
| - `ndcg@k` (predictive) → captured but **not auto-scored** unless `--include-predictive` | |
| (those train real models, slow). Plan precision/recall computed for every row. | |
| Run: | |
| ```bash | |
| export SDK_ACCESS_TOKEN=… LEXSI_WORKSPACE_NAME=… LEXSI_TEXT_PROJECT_NAME=… # + org/model/provider | |
| uv run python -m benchmark.pkdd_runner --llm lexsi --log-dir artifacts/pkdd_runs | |
| # scope / debug | |
| uv run python -m benchmark.pkdd_runner --tier analytic --limit 5 --llm lexsi | |
| uv run python -m benchmark.pkdd_runner --include-predictive --llm lexsi | |
| ``` | |
| Flags: `--tier`, `--include-predictive`, `--limit`, `--llm`, `--log-dir`, | |
| `--llm-cache-dir`/`--no-llm-cache`, `-v`. Outputs: per-row trace JSON + a per-tier | |
| scoreboard (pass/scored + plan p/r). The agent queries the fixture (handle from | |
| `load_pkdd_handle()` with `duckdb_path` repointed to the fixture). | |
| > Note: `bench/pkdd/sessions.yaml` (multi-turn) and `p2_feature_engineering.yaml` | |
| > (separate `assertions` schema) are **not** run by `pkdd_runner` yet. | |
| ### 13c. Datagen pipeline (generates the PKDD questions) — `benchmark/datagen/` | |
| Template-based; authoring is per-template, so ~10 templates yield 500+ rows. Stages: | |
| column inventory → templates → instantiate → execute gold SQL on the pinned fixture → | |
| LLM round-trip verify → predictive enrichment → multi-turn compose → emit YAML. | |
| ```bash | |
| uv run python -m benchmark.datagen pin-fixture # snapshot artifacts/financial.duckdb | |
| uv run python -m benchmark.datagen run --skip-roundtrip # fast (skip LLM Stage 4) | |
| uv run python -m benchmark.datagen run # full (needs Lexsi text env) | |
| ``` | |
| Every emitted row carries `fixture_sha`; a scoreboard from a different sha isn't comparable. | |
| The grading design (5 axes, tiers, pass^k) lives in [`v1_benchmarking.md`](v1_benchmarking.md). | |
| ### 13d. SDK repro / probe scripts — `scripts/repro_*` | |
| Standalone, self-contained scripts that hit the live Lexsi SDK against a real | |
| trained model and report OK/FAIL per call. They're how we validate the | |
| SDK-backed tools and file platform bugs. Each catches every call so one run | |
| produces a complete report; each leaves its project on the platform for | |
| Activity-Log inspection. | |
| - **`scripts/repro_tabicl/`** — reproduces TabICL `model_inference` + per-case XAI | |
| failures (empty `Exception` on T4 pods, inference hang/timeout, `case_predict` | |
| `ValidationError`). Ships its own PKDD CSVs in `data/`. | |
| - **`scripts/repro_model_lifecycle/`** — probe harness for the **model-lifecycle | |
| tools** (`list_available_models`, `select_active_model`, `compare_models`, | |
| `check_drift`, `monitor_performance`) + the future explanation methods. Trains | |
| TabICL + XGBoost on the (local) PKDD CSVs and calls each SDK method. Two scripts: | |
| `repro_model_lifecycle.py` (TabICL-first, the clean path) and | |
| `repro_model_lifecycle_xgboost_first.py` (XGBoost-first, reproduces the `#05-013` | |
| cascade). Run: `uv run python scripts/repro_model_lifecycle/repro_model_lifecycle.py` | |
| (env: `SDK_ACCESS_TOKEN`, `LEXSI_WORKSPACE_NAME`, `LEXSI_ORG_NAME`). | |
| **Live SDK findings from the lifecycle repro (last run 13 OK / 4 FAIL):** | |
| - `#05-013` — classic-ML (XGBoost) explainability scans **every stored column, | |
| including the server-injected `tag` column**, and fails `float()` on a | |
| non-numeric one. The SDK uploads clean data + an explicit `feature_include` | |
| (`tabular.py:285,368`); explainability ignores it. Model still **builds** | |
| (non-fatal), but `upload_data` raises and **rolls back the upload** → cascade | |
| (`train_model` → "Upload files first", predict upload → "Project Config is | |
| required"). TabICL's foundation path doesn't trip it. **Fix for the agent:** | |
| treat `#05-013` as success-with-warning, and train foundation-first / drop or | |
| encode non-numeric columns. | |
| - `evals_tabular(<foundation model>)` → `Exception: 'model'` (KeyError) — works | |
| for classic ML, breaks for TabICL. Blocks `compare_models` on foundation models. | |
| - `get_model_performance(...)` → `'utf-8' codec can't decode …` — response decode | |
| bug. Blocks `monitor_performance`. | |
| - per-case XAI (`case_predict`, `get_feature_importance`) needs (a) the model | |
| **active** + an inference run first (else "Inference status inactive"), and | |
| (b) SHAP **computed at train** (`xai=["shap"]`) — which itself trips `#05-013` | |
| for classic ML. `case_predict(risk_policies=True)` works without SHAP. | |
| The five lifecycle tools degrade gracefully against these (per-model error capture | |
| in `compare_models`; actionable `ok=False` in `monitor_performance`). Record new | |
| findings in [`sdk_issues.md`](sdk_issues.md). | |
| --- | |
| ## 14. Deploy | |
| - **Docker** ([Dockerfile](../Dockerfile)): `python:3.10-slim`, CPU-only torch, pinned | |
| deps, **bakes** `artifacts/financial.duckdb` (the PKDD source TSVs are Git-LFS | |
| pointers), serves Gradio on **7850**. Build for the platform arch: | |
| `docker buildx build --platform linux/amd64 --provenance=false --sbom=false -t bplexsi/lexsi-ds-agent:prod_vN --push .` | |
| - **HF Space** — a Gradio Space (`README.md` frontmatter `sdk: gradio`, | |
| `app_file: app/gradio_app.py`); the Dockerfile is ignored there. | |
| - **Lexsi platform** — [config.yaml](../config.yaml): one container on port 7850; the | |
| real `SDK_ACCESS_TOKEN` goes in the platform env tab (the repo keeps a placeholder). | |
| --- | |
| ## 15. Sharp edges (things that bit us) | |
| - **No LLM timeout = frozen run.** Always keep `_call_with_timeout` in the path (§9). | |
| - **Tool arg-name drift.** The planner reuses sibling names (`table_or_label`, | |
| `source_columns`, `prompt`); mismatched schemas → `validation_error` loops. Alias them. | |
| - **DataFrame truthiness.** `x = cache.get(a) or cache.get(b)` raises on a DataFrame; use explicit `is None`. | |
| - **String-typed dates/numbers.** Source columns are often VARCHAR; rely on the | |
| `ColumnInfo.sample` + the `TRY_CAST`/`try_strptime` guidance in the text_to_sql prompt. | |
| - **DB cache contamination.** The master must stay pristine — agents write to the | |
| working copy, never the master (§7). | |
| - **`artifacts/sessions/` growth.** Working copies/pgdata are cleaned now; if you add a | |
| loader, clean up after it. | |
| - **Single-run bench numbers are noisy** (DAB sits in an 8–9/17 band, 11/17 ceiling). | |
| Use `--iterations` and read the per-task/failure breakdown, not the headline. | |
| --- | |
| ## 16. "Where do I look for…" | |
| | Want to… | Go to | | |
| |---|---| | |
| | change the planner behavior / rules | `prompts.py` (`PLANNER_SYSTEM`) | | |
| | add/modify a tool | `tools/<name>.py` + `tools/__init__.py` (+ `_TOOL_GROUPS`) | | |
| | change which tools the planner sees | `prompts.py` selector (`_CORE_TOOLS`, `_TOOL_GROUPS`) | | |
| | connect a new datasource | `datasource.py` (loaders) + `connect_datalake` tool | | |
| | change LLM / caching / timeout | `llm/client.py` | | |
| | run/extend DAB | `benchmark/dab/{adapter,runner}.py` | | |
| | run/extend PKDD | `benchmark/pkdd_runner.py` + `bench/pkdd/questions.yaml` | | |
| | regenerate PKDD questions | `benchmark/datagen/` | | |
| | model-lifecycle tools (list/select/compare/drift/monitor) | `tools/{list_available_models,select_active_model,compare_models,check_drift,monitor_performance}.py` + `tools/_tabular_common.py` | | |
| | probe the live SDK / file an SDK bug | `scripts/repro_model_lifecycle/`, `scripts/repro_tabicl/` → `docs/sdk_issues.md` | | |
| | the UI | `app/gradio_app.py` | | |
| | roadmap / what's next | `v1_tools.md`, `tools_remaining_and_dab_hardening.md` | | |