docs/sdk_issues.md

Lexsi SDK — Issues, Workarounds, and Suggested Fixes

Compiled while building the Lexsi DS Agent (v0). All issues hit with lexsi-sdk==0.1.46 (PyPI, May 2026). Repo: Lexsi-Labs/Lexsi-sdk. Docs: sdk.lexsi.ai.

Severity tags:

• 🐛 Bug — reproducible code defect
• 📖 Documentation — code is fine, docs are wrong/misleading
• 🧩 Design — works as designed but creates a UX cliff for consumers
• ❓ Hypothesis — observed behavior matches this but we don't have a direct confirmation from the SDK code

Each issue records: where it lives in the SDK, our user-visible symptom, the workaround we landed in lexsi_ds_agent, and a suggested SDK-side fix where applicable.

---

1. 🐛 UnboundLocalError in TabularProject.model_inference when pod=None — ✅ RESOLVED upstream

Status: Fixed in lexsi-sdk 0.1.51 via PR #69. The new SDK makes pod a required positional argument on model_inference, eliminating the unbound-variable code path. Our predict tool still passes pod="small" by default, but it's now a required-arg shim rather than a bug-dodge.

Severity: Bug. Crashed on every call unless caller passes a non-None pod.

File: lexsi_sdk/core/tabular.py — inside TabularProject.model_inference.

Symptom seen from agent:

predict: model_inference failed: UnboundLocalError: local variable
'custom_batch_servers' referenced before assignment

Root cause — the validation block references custom_batch_servers in its else branch, but the variable is only bound inside the if branch:

if pod and self.metadata.get("modality") == "tabular":
    custom_batch_servers = self.api_client.get(AVAILABLE_BATCH_SERVERS_URI)
    available_custom_batch_servers = (
        custom_batch_servers.get("details", [])
        + custom_batch_servers.get("available_gpu_custom_servers", [])
    )
    Validate.value_against_list(
        "pod", pod,
        [server["instance_name"] for server in available_custom_batch_servers],
    )
else:
    Validate.value_against_list(
        "pod", pod,
        [
            server["instance_name"]
            # ↓ NameError: custom_batch_servers never bound on this path
            for server in custom_batch_servers.get("details", [])
        ],
    )

When pod=None (the default, the most common call pattern) or when the project's modality is non-tabular, control falls into the else branch and Python raises UnboundLocalError.

Our workaround: lexsi_ds/agent/tools/predict.py always passes pod="small" so we hit the if branch and dodge the unbound variable. See the _DEFAULT_PREDICT_POD constant.

Suggested fix — move the fetch out of the conditional and only validate when the caller actually passed a pod:

if pod is not None:
    custom_batch_servers = self.api_client.get(AVAILABLE_BATCH_SERVERS_URI)
    available_servers = custom_batch_servers.get("details", [])
    if self.metadata.get("modality") == "tabular":
        available_servers = (
            available_servers
            + custom_batch_servers.get("available_gpu_custom_servers", [])
        )
    Validate.value_against_list(
        "pod", pod,
        [server["instance_name"] for server in available_servers],
    )
# else: pod is None, server uses its default, no validation needed

Three-line patch. Suggested PR title: fix(tabular): UnboundLocalError when model_inference is called without pod.

---

2. 🐛 ProjectConfig.pred_label=None crashes the server-side training pipeline

Severity: Bug. Every upload_data(config=ProjectConfig(...)) call with no pred_label lands here.

Files:

• Type declared with Optional[str] in lexsi_sdk/common/types.py (class ProjectConfig).
• SDK forwards config.get("pred_label") as-is in lexsi_sdk/core/tabular.py → upload_data payload build.

Symptom seen from agent: Activity Log shows

Started building XGBoost_default model
Building Model
'pred_label'
Failed while running explainability. #05-013

or for foundation models:

Started building TabPFN_v3 model
Building Model
'pred_label'
Failed while building model. #05-001

The bare 'pred_label' log line is a Python KeyError/NoneType access being logged. The wrapping step labels the failure as #05-013 (XAI) or #05-001 (model build) depending on where the access happens.

Root cause: the server-side handler does direct subscript access like cfg["pred_label"] or attribute access without first checking for None. Even though the type is documented as Optional[str], the server can't actually accept None.

Our workaround: Set pred_label="Prediction" unconditionally in _build_project_config (lexsi_ds/agent/tools/train_tabular_model.py). Combined with workaround #3 below, training succeeds.

Suggested fix: server-side handler should cfg.get("pred_label") and gate any column-name logic on a non-None value. If pred_label is genuinely required for the pipeline, change the docstring + TypedDict to remove Optional and make this a validation error at SDK level, not a server crash.

3. 🧩 No way to delete or update a saved project config (update_config is commented-out dead code)

Severity: Design — forces consumers to use one-project-per-run.

File: lexsi_sdk/core/project.py — def update_config exists but the entire body is commented out:

def update_config(self, compute_type: str, config: DataConfig) -> str:
    #     """Update the project configurations. Accepts a config dictionary.
    #     ...
    #     """
    #     if not config:
    #         raise Exception("Please upload config")
    #     ...

Symptom seen from agent: After the first successful upload_data(config=…), the project has a saved config. On the second call:

train_tabular_model: upload_data failed: Exception: Config already exists, please remove config

The SDK source confirms:

if project_config != "Not Found" and config:
    raise Exception("Config already exists, please remove config")

…and there's no method to "remove" it. Documentation says to "update" but the implementation is dead.

Our workaround: Create a fresh per-run Lexsi project via Workspace.create_project(name="agent<task><runid>", modality="tabular", project_type=<task>). Every agent run gets a clean slate. See _get_or_create_run_project in train_tabular_model.py. This works but accumulates projects on the Lexsi side that the user manually cleans up — there's no Workspace.delete_project (only Project.delete_project, which works but isn't exposed in our cleanup flow yet).

Suggested fixes:

• Resurrect update_config — finish the implementation that's commented out. The endpoint clearly existed at some point.
• OR add Project.delete_config — single-purpose, "wipe my saved config so I can upload a new one". Maps to whatever DELETE endpoint exists.
• OR allow upload_data(config=...) to replace an existing config when the caller passes force=True or similar.

---

4. 🐛 Server-side runs XAI by default even when xai_method is absent / empty

Severity: Bug — or undocumented behavior, depending on intent.

Files: SDK side is correct — upload_data only includes explainability_method in the payload when truthy:

if config.get("xai_method"):
    payload["metadata"]["explainability_method"] = config.get("xai_method")

So passing xai_method=[] or omitting the key should mean "don't run XAI." But:

Symptom seen from agent: We tried xai_method=[] and the server still ran XAI and crashed (#05-013 Failed while running explainability with could not convert string to float: 'class_0' in the Activity Log).

Hypothesis: the server-side training pipeline runs SHAP by default whenever a classification model is trained, regardless of what the upload metadata says. There's no platform-level "don't run XAI" flag we could find.

Our workaround: train_tabular_model.py's xai_method default is []. It's a no-op against the current server behavior but documents intent and is the right path forward if/when the server is fixed.

Suggested fixes:

• Make the server honor "no XAI requested" explicitly: when explainability_method is absent OR empty list, skip the XAI step.
• Add an explicit skip_xai: bool flag on ProjectConfig for unambiguous opt-out.
• OR document the current behavior in the ProjectConfig docstring: "If absent or empty, server defaults to ['shap'] for classification and ['shap'] for regression."

---

5. 🧩 xai_method=['lime'] "not available for classic models default version" — tier gate not surfaced upfront

Severity: Design — error message is reasonable but consumers can't discover the constraint via the SDK before making the call.

Symptom seen from agent:

Lime is not available for classic models default version.

when we tried xai_method=["lime"] with model_type="XGBoost".

The four valid XAI methods per lexsi_sdk/core/tabular.py validation: ["shap", "lime", "ig", "dlb"]. But LIME requires a higher Lexsi tier when used with classic ML (XGBoost, LGBoost, etc.), and IG / DLB are deep-learning-only. So in practice on the default tier, SHAP is the only working XAI for classic ML.

Our workaround: Default to xai_method=[] (skip training-time XAI entirely, see issue #6).

Suggested fixes:

• Per-tier capability endpoint: LEXSI.available_xai_methods(model_type) → ["shap"] on default, ["shap", "lime"] on premium, etc. So SDK can validate before the round-trip.
• Document the tier matrix in xai_method's docstring on ProjectConfig.

---

6. 🐛 TabPFN inference fails with CUDA-on-CPU mismatch (#05-004)

Severity: Bug — server-side or compute-pod-config issue.

Symptom seen from agent: After successfully training TabPFN on compute_type="T4.small" (GPU), predict fails with:

Model Inference Failed. #05-004

Activity Log details:

Attempting to deserialize object on a CUDA device but
torch.cuda.is_available() is False. If you are running on a CPU-only
machine, please use torch.load with map_location=torch.device('cpu')
to map your storages to the CPU.

Root cause: The TabPFN model checkpoint was saved with CUDA tensors. The inference pod Lexsi spun up doesn't have a GPU, but the loader does torch.load(..., map_location=DEFAULT) instead of map_location='cpu'. PyTorch refuses to deserialize.

Our workaround: None really. We currently pass pod="small" (CPU) to predict to dodge the UnboundLocalError from issue #1. The user is steered to use XGBoost rather than TabPFN until this is fixed.

Suggested fixes:

• Inference-side fix: when loading a model checkpoint, use map_location='cpu' if not torch.cuda.is_available() else None so a CUDA-saved model can be loaded on a CPU pod.
• Or: route TabPFN inference to a GPU pod automatically when the model was trained on one — model_inference could look at models()['compute_type'] and reject CPU pods for GPU-trained models with a clearer error.

---

7. 🐛 case_predict can't parse its own response for an unlabeled (prediction) case — CaseTabular.true_value is required

Severity: Bug — breaks explain_prediction entirely for prediction-tag cases (the common case: explaining a row whose outcome isn't known yet).

File: lexsi_sdk/core/tabular.py — the CaseTabular pydantic model. On lexsi-sdk==0.1.51:

CaseTabular.true_value:    str | int   # required=True, no default
CaseTabular.pred_value:    str | int   # required=True
CaseTabular.pred_category: str | int   # required=True

Symptom seen from the repro script (case_predict(unique_identifier='5494', tag=<predict_tag>, model_name='TabICL_v1', xai=['shap']) on a tag with no ground-truth column):

FAIL case_predict in 36.5s. ValidationError: 2 validation errors for CaseTabular
true_value.str
  Input should be a valid string [input_value=None, input_type=NoneType]
true_value.int
  Input should be a valid integer [input_value=None, input_type=NoneType]

(It's one field, not two errors — pydantic reports both arms of the str | int union failing against None.)

Root cause: a case from a predict tag has no observed label, so the server legitimately returns true_value = null. But the response model types true_value as a required str | int, so pydantic raises while deserializing — case_predict blows up before it can return the CaseTabular. The agent's explain_prediction tool calls case_predict on exactly these unlabeled rows, so every case fails → all_cases_failed.

Our workaround (shipped): explain_prediction._relax_casetabular_validation() relaxes true_value / pred_value / pred_category to Optional[str | int] at runtime (pydantic-v2 model_fields patch + model_rebuild(force=True)) before calling case_predict, so unlabeled prediction cases parse. It's idempotent and best-effort (guarded: if the SDK internals change, the original error surfaces). Remove once the SDK ships the fix.

Suggested fix (SDK): make the label-dependent fields optional on the response model — true_value: Optional[str | int] = None (and pred_value / pred_category, which are also null for unscored cases). A case without ground truth should parse with true_value=None, not raise.

Note (attribute surface — verified, no mismatch): CaseTabular does expose the fields explain_prediction.py reads — shap_feature_importance (Optional[Dict]), pred_value, pred_category, summary — plus methods xai_shap(), xai_summary(), xai_similar_cases(). (feature_importance(feature: str) -> float is a per-feature method, not the dict.) So once the validation bug above is worked around, the tool reads the right surface.