| --- |
| title: EDAgent |
| emoji: 📊 |
| colorFrom: blue |
| colorTo: indigo |
| sdk: gradio |
| app_file: app.py |
| pinned: false |
| short_description: A 1.5B agent that writes, runs and repairs its own EDA code |
| --- |
| |
| # 📊 EDAgent |
|
|
| Point it at any Hugging Face dataset. It extracts the schema, writes pandas / |
| matplotlib analysis code, executes that code, repairs it from the traceback if it |
| crashes, and assembles a Markdown + PDF report with tables and plots. |
|
|
| Built on **Qwen2.5-Coder-1.5B-Instruct**, small enough to run on a free GPU. |
|
|
| Three **Quick Starters** are pre-generated and load instantly without touching the |
| model, so the app is usable even when a visitor's daily GPU quota is exhausted. |
|
|
| ## Architecture |
|
|
| | Stage | Owner | What happens | |
| |---|---|---| |
| | 1. Load + normalize | Python | `load_dataset` with a raw-file fallback; quoted nulls converted to real `NaN`; index-like columns dropped | |
| | 2. Context card | Python | One line per column: dtype, null count, cardinality, range | |
| | 3. Code generation | **LLM** | Schema + instruction → one fenced Python block | |
| | 4. Execution | Python | `exec` in an isolated namespace, stdout captured, plots collected | |
| | 5. Repair (×1) | **LLM** | Cleaned traceback (offending line + message) → corrected script | |
| | 6. Facts + tables | Python | `idxmax`, `describe`, `corr` → Markdown tables | |
| | 7. Narrative | **LLM** | One sentence per pre-computed fact | |
|
|
| ## Design note |
|
|
| The model owns two jobs: writing analysis code and repairing it. Everything else — |
| table rendering, plot placement, and every "which is highest" lookup — is computed |
| in Python and handed to the model as a stated fact. |
|
|
| This was arrived at empirically. A 1.5B model reliably produced correct *descriptions* |
| and unreliable *lookups*: it rotated group means against their labels, invented |
| correlation pairs from fragments of two real ones, and inverted orderings. Adding |
| rules to the prompt made compliance worse, not better — at six simultaneous |
| constraints it stopped writing prose and began transcribing the input's structure. |
| Every fix that worked removed responsibility from the model rather than adding |
| instruction to it. |
|
|
| `df.corr()` on mixed-type data is patched deterministically rather than left to the |
| repair loop: it failed on both test datasets, and the model fixed it correctly once |
| (`select_dtypes(...).corr()`) and incorrectly once (`dropna(subset=[...]).corr()`). |
|
|
| ## Plot and table quality |
|
|
| Generated plotting code is wrapped rather than trusted. `seaborn.heatmap`, |
| `seaborn.histplot` and `plt.savefig` are patched around execution so that, on any |
| dataset: |
|
|
| - correlation heatmaps scale their canvas with the matrix and drop cell annotations |
| past 12 columns; |
| - long column names are abbreviated while keeping their suffixes; |
| - axes with more than 20 tick labels are thinned; |
| - heavily right-skewed distributions (prices, fares, incomes) are replotted on a log |
| axis; |
| - no axis or table ever shows scientific notation — `37,000,000`, not `3.7e+07`. |
|
|
| Near-perfect correlations (|r| ≥ 0.99) are separated into a **Redundant Columns** |
| section, since they are almost always duplicate encodings rather than findings. |
|
|
| ## Known limitations |
|
|
| - The model is text-only and never sees the plots it generates; it is explicitly |
| prevented from describing them. |
| - `audit_numbers` flags numerals absent from the evidence. It detects fabrication, |
| not misinterpretation, and false-positives on derived values. |
| - The model occasionally embellishes a supplied fact. This is the residual failure |
| mode once every lookup has been moved into Python. |
| - Non-tabular, image and audio datasets are not supported. |
| - One repair attempt only. If it fails, the report is built from the data alone. |
|
|
| ## Security |
|
|
| This Space executes LLM-generated Python. A static check blocks `subprocess`, |
| `socket`, `open()`, `eval`/`exec` and `os`/`sys` imports, but `exec` is not a |
| sandbox — treat the guard as a speed bump. **Do not add secrets to this Space's |
| settings.** |
|
|
| ## Hardware |
|
|
| Built for **ZeroGPU**. The model is placed on `cuda` at module level and all GPU work |
| runs inside a `@spaces.GPU` function; the app also falls back to CPU so it boots |
| anywhere. One fresh report is about seven generation calls, roughly 60–90 s of GPU |
| time. Results are cached in memory, and the Quick Starters are served from disk, so |
| repeat views cost nothing. |
|
|
| ## Repository layout |
|
|
| ``` |
| app.py the agent and the Gradio UI |
| requirements.txt |
| quickstarts/ |
| titanic/ meta.json report.md code.py stdout.txt report.pdf plots/*.png |
| iris/ ... |
| housing/ ... |
| ``` |
|
|
| `quickstarts/` is generated by the companion Colab notebook. If it is absent, the |
| Quick Starter buttons fall back to a live run. |
