docs/sdk_issues.md

# Lexsi SDK — Issues, Workarounds, and Suggested Fixes

Compiled while building the Lexsi DS Agent (v0). All issues hit with
`lexsi-sdk==0.1.46` (PyPI, May 2026). Repo:
[Lexsi-Labs/Lexsi-sdk](https://github.com/Lexsi-Labs/Lexsi-sdk). Docs:
[sdk.lexsi.ai](https://sdk.lexsi.ai/).

Severity tags:
- 🐛 **Bug** — reproducible code defect
- 📖 **Documentation** — code is fine, docs are wrong/misleading
- 🧩 **Design** — works as designed but creates a UX cliff for consumers
- ❓ **Hypothesis** — observed behavior matches this but we don't have a
  direct confirmation from the SDK code

Each issue records: where it lives in the SDK, our user-visible symptom,
the workaround we landed in `lexsi_ds_agent`, and a suggested SDK-side
fix where applicable.

---

## 1. 🐛 `UnboundLocalError` in `TabularProject.model_inference` when `pod=None` — ✅ RESOLVED upstream

**Status:** Fixed in [lexsi-sdk 0.1.51](https://github.com/Lexsi-Labs/Lexsi-sdk/releases/tag/0.1.51) via [PR #69](https://github.com/Lexsi-Labs/Lexsi-sdk/pull/69). The new SDK makes `pod` a required positional argument on `model_inference`, eliminating the unbound-variable code path. Our `predict` tool still passes `pod="small"` by default, but it's now a required-arg shim rather than a bug-dodge.

**Severity:** Bug. **Crashed on every call** unless caller passes a non-None `pod`.

**File:** [`lexsi_sdk/core/tabular.py`](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/core/tabular.py) — inside `TabularProject.model_inference`.

**Symptom seen from agent:**
```
predict: model_inference failed: UnboundLocalError: local variable
'custom_batch_servers' referenced before assignment
```

**Root cause** — the validation block references `custom_batch_servers`
in its `else` branch, but the variable is only bound inside the `if`
branch:

```python
if pod and self.metadata.get("modality") == "tabular":
    custom_batch_servers = self.api_client.get(AVAILABLE_BATCH_SERVERS_URI)
    available_custom_batch_servers = (
        custom_batch_servers.get("details", [])
        + custom_batch_servers.get("available_gpu_custom_servers", [])
    )
    Validate.value_against_list(
        "pod", pod,
        [server["instance_name"] for server in available_custom_batch_servers],
    )
else:
    Validate.value_against_list(
        "pod", pod,
        [
            server["instance_name"]
            # ↓ NameError: custom_batch_servers never bound on this path
            for server in custom_batch_servers.get("details", [])
        ],
    )
```

When `pod=None` (the default, the most common call pattern) or when the
project's modality is non-tabular, control falls into the `else` branch
and Python raises `UnboundLocalError`.

**Our workaround:** [`lexsi_ds/agent/tools/predict.py`](../lexsi_ds/agent/tools/predict.py) always passes `pod="small"` so we hit the `if` branch and dodge the unbound variable. See the `_DEFAULT_PREDICT_POD` constant.

**Suggested fix** — move the fetch out of the conditional and only validate when the caller actually passed a `pod`:

```python
if pod is not None:
    custom_batch_servers = self.api_client.get(AVAILABLE_BATCH_SERVERS_URI)
    available_servers = custom_batch_servers.get("details", [])
    if self.metadata.get("modality") == "tabular":
        available_servers = (
            available_servers
            + custom_batch_servers.get("available_gpu_custom_servers", [])
        )
    Validate.value_against_list(
        "pod", pod,
        [server["instance_name"] for server in available_servers],
    )
# else: pod is None, server uses its default, no validation needed
```

Three-line patch. Suggested PR title: `fix(tabular): UnboundLocalError when model_inference is called without pod`.

---

## 2. 🐛 `ProjectConfig.pred_label=None` crashes the server-side training pipeline

**Severity:** Bug. Every `upload_data(config=ProjectConfig(...))` call with no `pred_label` lands here.

**Files:**
- Type declared with `Optional[str]` in [`lexsi_sdk/common/types.py`](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/common/types.py) (`class ProjectConfig`).
- SDK forwards `config.get("pred_label")` as-is in
  [`lexsi_sdk/core/tabular.py`](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/core/tabular.py) → `upload_data` payload build.

**Symptom seen from agent:** Activity Log shows
```
Started building XGBoost_default model
Building Model
'pred_label'
Failed while running explainability. #05-013
```
or for foundation models:
```
Started building TabPFN_v3 model
Building Model
'pred_label'
Failed while building model. #05-001
```

The bare `'pred_label'` log line is a Python `KeyError`/`NoneType` access being logged. The wrapping step labels the failure as `#05-013` (XAI) or `#05-001` (model build) depending on where the access happens.

**Root cause:** the server-side handler does direct subscript access like `cfg["pred_label"]` or attribute access without first checking for `None`. Even though the type is documented as `Optional[str]`, the server can't actually accept `None`.

**Our workaround:** Set `pred_label="Prediction"` unconditionally in
`_build_project_config` ([`lexsi_ds/agent/tools/train_tabular_model.py`](../lexsi_ds/agent/tools/train_tabular_model.py)). Combined with workaround #3 below, training succeeds.

**Suggested fix:** server-side handler should `cfg.get("pred_label")` and gate any column-name logic on a non-None value. If pred_label is genuinely required for the pipeline, change the docstring + TypedDict to remove `Optional` and make this a validation error at SDK level, not a server crash.


## 3. 🧩 No way to delete or update a saved project config (`update_config` is commented-out dead code)

**Severity:** Design — forces consumers to use one-project-per-run.

**File:** [`lexsi_sdk/core/project.py`](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/core/project.py) — `def update_config` exists but the entire body is commented out:

```python
def update_config(self, compute_type: str, config: DataConfig) -> str:
    #     """Update the project configurations. Accepts a config dictionary.
    #     ...
    #     """
    #     if not config:
    #         raise Exception("Please upload config")
    #     ...
```

**Symptom seen from agent:** After the first successful `upload_data(config=…)`, the project has a saved config. On the second call:
```
train_tabular_model: upload_data failed: Exception: Config already exists, please remove config
```
The SDK source confirms:
```python
if project_config != "Not Found" and config:
    raise Exception("Config already exists, please remove config")
```
…and there's no method to "remove" it. Documentation says to "update" but the implementation is dead.

**Our workaround:** Create a fresh per-run Lexsi project via `Workspace.create_project(name="agent<task><runid>", modality="tabular", project_type=<task>)`. Every agent run gets a clean slate. See `_get_or_create_run_project` in `train_tabular_model.py`. This works but accumulates projects on the Lexsi side that the user manually cleans up — there's no `Workspace.delete_project` (only `Project.delete_project`, which works but isn't exposed in our cleanup flow yet).

**Suggested fixes:**
- **Resurrect `update_config`** — finish the implementation that's commented out. The endpoint clearly existed at some point.
- **OR add `Project.delete_config`** — single-purpose, "wipe my saved config so I can upload a new one". Maps to whatever DELETE endpoint exists.
- **OR allow `upload_data(config=...)` to *replace* an existing config** when the caller passes `force=True` or similar.

---

## 4. 🐛 Server-side runs XAI by default even when `xai_method` is absent / empty

**Severity:** Bug — or undocumented behavior, depending on intent.

**Files:** SDK side is correct — `upload_data` only includes `explainability_method` in the payload when truthy:
```python
if config.get("xai_method"):
    payload["metadata"]["explainability_method"] = config.get("xai_method")
```
So passing `xai_method=[]` or omitting the key should mean "don't run XAI." But:

**Symptom seen from agent:** We tried `xai_method=[]` and the server still ran XAI and crashed (`#05-013 Failed while running explainability` with `could not convert string to float: 'class_0'` in the Activity Log).

**Hypothesis:** the server-side training pipeline runs SHAP by default whenever a classification model is trained, regardless of what the upload metadata says. There's no platform-level "don't run XAI" flag we could find.

**Our workaround:** [`train_tabular_model.py`](../lexsi_ds/agent/tools/train_tabular_model.py)'s `xai_method` default is `[]`. It's a no-op against the current server behavior but documents intent and is the right path forward if/when the server is fixed.

**Suggested fixes:**
- Make the server honor "no XAI requested" explicitly: when `explainability_method` is absent OR empty list, skip the XAI step.
- Add an explicit `skip_xai: bool` flag on `ProjectConfig` for unambiguous opt-out.
- OR document the current behavior in the [ProjectConfig docstring](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/common/types.py): "If absent or empty, server defaults to `['shap']` for classification and `['shap']` for regression."

---

## 5. 🧩 `xai_method=['lime']` "not available for classic models default version" — tier gate not surfaced upfront

**Severity:** Design — error message is reasonable but consumers can't discover the constraint via the SDK before making the call.

**Symptom seen from agent:**
```
Lime is not available for classic models default version.
```
when we tried `xai_method=["lime"]` with `model_type="XGBoost"`.

**The four valid XAI methods** per [`lexsi_sdk/core/tabular.py`](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/core/tabular.py) validation: `["shap", "lime", "ig", "dlb"]`. But LIME requires a higher Lexsi tier when used with classic ML (XGBoost, LGBoost, etc.), and IG / DLB are deep-learning-only. So in practice on the default tier, **SHAP is the only working XAI for classic ML.**

**Our workaround:** Default to `xai_method=[]` (skip training-time XAI entirely, see issue #6).

**Suggested fixes:**
- **Per-tier capability endpoint:** `LEXSI.available_xai_methods(model_type)` → `["shap"]` on default, `["shap", "lime"]` on premium, etc. So SDK can validate before the round-trip.
- **Document the tier matrix** in `xai_method`'s docstring on `ProjectConfig`.

---

## 6. 🐛 TabPFN inference fails with CUDA-on-CPU mismatch (`#05-004`)

**Severity:** Bug — server-side or compute-pod-config issue.

**Symptom seen from agent:** After successfully training TabPFN on `compute_type="T4.small"` (GPU), `predict` fails with:
```
Model Inference Failed. #05-004
```
Activity Log details:
```
Attempting to deserialize object on a CUDA device but
torch.cuda.is_available() is False. If you are running on a CPU-only
machine, please use torch.load with map_location=torch.device('cpu')
to map your storages to the CPU.
```

**Root cause:** The TabPFN model checkpoint was saved with CUDA tensors. The inference pod Lexsi spun up doesn't have a GPU, but the loader does `torch.load(..., map_location=DEFAULT)` instead of `map_location='cpu'`. PyTorch refuses to deserialize.

**Our workaround:** None really. We currently pass `pod="small"` (CPU) to `predict` to dodge the `UnboundLocalError` from issue #1. The user is steered to use XGBoost rather than TabPFN until this is fixed.

**Suggested fixes:**
- **Inference-side fix:** when loading a model checkpoint, use `map_location='cpu' if not torch.cuda.is_available() else None` so a CUDA-saved model can be loaded on a CPU pod.
- **Or:** route TabPFN inference to a GPU pod automatically when the model was trained on one — `model_inference` could look at `models()['compute_type']` and reject CPU pods for GPU-trained models with a clearer error.

---

## 7. 🐛 `case_predict` can't parse its own response for an unlabeled (prediction) case — `CaseTabular.true_value` is required

**Severity:** Bug — **breaks `explain_prediction` entirely** for prediction-tag cases (the common case: explaining a row whose outcome isn't known yet).

**File:** [`lexsi_sdk/core/tabular.py`](https://github.com/Lexsi-Labs/Lexsi-sdk/blob/main/lexsi_sdk/core/tabular.py) — the `CaseTabular` pydantic model. On `lexsi-sdk==0.1.51`:

```python
CaseTabular.true_value:    str | int   # required=True, no default
CaseTabular.pred_value:    str | int   # required=True
CaseTabular.pred_category: str | int   # required=True
```

**Symptom seen from the repro script** (`case_predict(unique_identifier='5494', tag=<predict_tag>, model_name='TabICL_v1', xai=['shap'])` on a tag with no ground-truth column):

```
FAIL case_predict in 36.5s. ValidationError: 2 validation errors for CaseTabular
true_value.str
  Input should be a valid string [input_value=None, input_type=NoneType]
true_value.int
  Input should be a valid integer [input_value=None, input_type=NoneType]
```

(It's **one** field, not two errors — pydantic reports both arms of the `str | int` union failing against `None`.)

**Root cause:** a case from a **predict tag has no observed label**, so the server legitimately returns `true_value = null`. But the response model types `true_value` as a *required* `str | int`, so pydantic raises while deserializing — `case_predict` blows up before it can return the `CaseTabular`. The agent's [`explain_prediction`](../lexsi_ds/agent/tools/explain_prediction.py) tool calls `case_predict` on exactly these unlabeled rows, so **every case fails** → `all_cases_failed`.

**Our workaround (shipped):** `explain_prediction._relax_casetabular_validation()` relaxes `true_value` / `pred_value` / `pred_category` to `Optional[str | int]` at runtime (pydantic-v2 `model_fields` patch + `model_rebuild(force=True)`) before calling `case_predict`, so unlabeled prediction cases parse. It's idempotent and best-effort (guarded: if the SDK internals change, the original error surfaces). Remove once the SDK ships the fix.

**Suggested fix (SDK):** make the label-dependent fields optional on the response model —
`true_value: Optional[str | int] = None` (and `pred_value` / `pred_category`, which are also `null` for unscored cases). A case without ground truth should parse with `true_value=None`, not raise.

**Note (attribute surface — verified, no mismatch):** `CaseTabular` *does* expose the fields `explain_prediction.py` reads — `shap_feature_importance` (`Optional[Dict]`), `pred_value`, `pred_category`, `summary` — plus methods `xai_shap()`, `xai_summary()`, `xai_similar_cases()`. (`feature_importance(feature: str) -> float` is a per-feature *method*, not the dict.) So once the validation bug above is worked around, the tool reads the right surface.