README.md

metadata

title: EDAgent
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: false
short_description: A 1.5B agent that writes, runs and repairs its own EDA code

📊 EDAgent

Point it at any Hugging Face dataset. It extracts the schema, writes pandas / matplotlib analysis code, executes that code, repairs it from the traceback if it crashes, and assembles a Markdown + PDF report with tables and plots.

Built on Qwen2.5-Coder-1.5B-Instruct, small enough to run on a free GPU.

Three Quick Starters are pre-generated and load instantly without touching the model, so the app is usable even when a visitor's daily GPU quota is exhausted.

Architecture

Stage	Owner	What happens
1. Load + normalize	Python	load_dataset with a raw-file fallback; quoted nulls converted to real NaN; index-like columns dropped
2. Context card	Python	One line per column: dtype, null count, cardinality, range
3. Code generation	LLM	Schema + instruction → one fenced Python block
4. Execution	Python	exec in an isolated namespace, stdout captured, plots collected
5. Repair (×1)	LLM	Cleaned traceback (offending line + message) → corrected script
6. Facts + tables	Python	idxmax, describe, corr → Markdown tables
7. Narrative	LLM	One sentence per pre-computed fact

Design note

The model owns two jobs: writing analysis code and repairing it. Everything else — table rendering, plot placement, and every "which is highest" lookup — is computed in Python and handed to the model as a stated fact.

This was arrived at empirically. A 1.5B model reliably produced correct descriptions and unreliable lookups: it rotated group means against their labels, invented correlation pairs from fragments of two real ones, and inverted orderings. Adding rules to the prompt made compliance worse, not better — at six simultaneous constraints it stopped writing prose and began transcribing the input's structure. Every fix that worked removed responsibility from the model rather than adding instruction to it.

df.corr() on mixed-type data is patched deterministically rather than left to the repair loop: it failed on both test datasets, and the model fixed it correctly once (select_dtypes(...).corr()) and incorrectly once (dropna(subset=[...]).corr()).

Plot and table quality

Generated plotting code is wrapped rather than trusted. seaborn.heatmap, seaborn.histplot and plt.savefig are patched around execution so that, on any dataset:

• correlation heatmaps scale their canvas with the matrix and drop cell annotations past 12 columns;
• long column names are abbreviated while keeping their suffixes;
• axes with more than 20 tick labels are thinned;
• heavily right-skewed distributions (prices, fares, incomes) are replotted on a log axis;
• no axis or table ever shows scientific notation — 37,000,000, not 3.7e+07.

Near-perfect correlations (|r| ≥ 0.99) are separated into a Redundant Columns section, since they are almost always duplicate encodings rather than findings.

Known limitations

• The model is text-only and never sees the plots it generates; it is explicitly prevented from describing them.
• audit_numbers flags numerals absent from the evidence. It detects fabrication, not misinterpretation, and false-positives on derived values.
• The model occasionally embellishes a supplied fact. This is the residual failure mode once every lookup has been moved into Python.
• Non-tabular, image and audio datasets are not supported.
• One repair attempt only. If it fails, the report is built from the data alone.

Security

This Space executes LLM-generated Python. A static check blocks subprocess, socket, open(), eval/exec and os/sys imports, but exec is not a sandbox — treat the guard as a speed bump. Do not add secrets to this Space's settings.

Hardware

Built for ZeroGPU. The model is placed on cuda at module level and all GPU work runs inside a @spaces.GPU function; the app also falls back to CPU so it boots anywhere. One fresh report is about seven generation calls, roughly 60–90 s of GPU time. Results are cached in memory, and the Quick Starters are served from disk, so repeat views cost nothing.

Repository layout

app.py                     the agent and the Gradio UI
requirements.txt
quickstarts/
  titanic/   meta.json report.md code.py stdout.txt report.pdf plots/*.png
  iris/      ...
  housing/   ...

quickstarts/ is generated by the companion Colab notebook. If it is absent, the Quick Starter buttons fall back to a live run.