README.md

---
title: EDAgent
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: false
short_description: A 1.5B agent that writes, runs and repairs its own EDA code
---

# 📊 EDAgent

Point it at any Hugging Face dataset. It extracts the schema, writes pandas /
matplotlib analysis code, executes that code, repairs it from the traceback if it
crashes, and assembles a Markdown + PDF report with tables and plots.

Built on **Qwen2.5-Coder-1.5B-Instruct**, small enough to run on a free GPU.

Three **Quick Starters** are pre-generated and load instantly without touching the
model, so the app is usable even when a visitor's daily GPU quota is exhausted.

## Architecture

| Stage | Owner | What happens |
|---|---|---|
| 1. Load + normalize | Python | `load_dataset` with a raw-file fallback; quoted nulls converted to real `NaN`; index-like columns dropped |
| 2. Context card | Python | One line per column: dtype, null count, cardinality, range |
| 3. Code generation | **LLM** | Schema + instruction → one fenced Python block |
| 4. Execution | Python | `exec` in an isolated namespace, stdout captured, plots collected |
| 5. Repair (×1) | **LLM** | Cleaned traceback (offending line + message) → corrected script |
| 6. Facts + tables | Python | `idxmax`, `describe`, `corr` → Markdown tables |
| 7. Narrative | **LLM** | One sentence per pre-computed fact |

## Design note

The model owns two jobs: writing analysis code and repairing it. Everything else —
table rendering, plot placement, and every "which is highest" lookup — is computed
in Python and handed to the model as a stated fact.

This was arrived at empirically. A 1.5B model reliably produced correct *descriptions*
and unreliable *lookups*: it rotated group means against their labels, invented
correlation pairs from fragments of two real ones, and inverted orderings. Adding
rules to the prompt made compliance worse, not better — at six simultaneous
constraints it stopped writing prose and began transcribing the input's structure.
Every fix that worked removed responsibility from the model rather than adding
instruction to it.

`df.corr()` on mixed-type data is patched deterministically rather than left to the
repair loop: it failed on both test datasets, and the model fixed it correctly once
(`select_dtypes(...).corr()`) and incorrectly once (`dropna(subset=[...]).corr()`).

## Plot and table quality

Generated plotting code is wrapped rather than trusted. `seaborn.heatmap`,
`seaborn.histplot` and `plt.savefig` are patched around execution so that, on any
dataset:

- correlation heatmaps scale their canvas with the matrix and drop cell annotations
  past 12 columns;
- long column names are abbreviated while keeping their suffixes;
- axes with more than 20 tick labels are thinned;
- heavily right-skewed distributions (prices, fares, incomes) are replotted on a log
  axis;
- no axis or table ever shows scientific notation — `37,000,000`, not `3.7e+07`.

Near-perfect correlations (|r| ≥ 0.99) are separated into a **Redundant Columns**
section, since they are almost always duplicate encodings rather than findings.

## Known limitations

- The model is text-only and never sees the plots it generates; it is explicitly
  prevented from describing them.
- `audit_numbers` flags numerals absent from the evidence. It detects fabrication,
  not misinterpretation, and false-positives on derived values.
- The model occasionally embellishes a supplied fact. This is the residual failure
  mode once every lookup has been moved into Python.
- Non-tabular, image and audio datasets are not supported.
- One repair attempt only. If it fails, the report is built from the data alone.

## Security

This Space executes LLM-generated Python. A static check blocks `subprocess`,
`socket`, `open()`, `eval`/`exec` and `os`/`sys` imports, but `exec` is not a
sandbox — treat the guard as a speed bump. **Do not add secrets to this Space's
settings.**

## Hardware

Built for **ZeroGPU**. The model is placed on `cuda` at module level and all GPU work
runs inside a `@spaces.GPU` function; the app also falls back to CPU so it boots
anywhere. One fresh report is about seven generation calls, roughly 60–90 s of GPU
time. Results are cached in memory, and the Quick Starters are served from disk, so
repeat views cost nothing.

## Repository layout

```
app.py                     the agent and the Gradio UI
requirements.txt
quickstarts/
  titanic/   meta.json report.md code.py stdout.txt report.pdf plots/*.png
  iris/      ...
  housing/   ...
```

`quickstarts/` is generated by the companion Colab notebook. If it is absent, the
Quick Starter buttons fall back to a live run.