Spaces:
Sleeping
Sleeping
| title: DataPilot AI Agent | |
| emoji: "📊" | |
| colorFrom: indigo | |
| colorTo: green | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: true | |
| license: mit | |
| short_description: Evidence-grounded autonomous data science and ML copilot | |
| <!-- Hugging Face production deployment: bounded Gemini REST and server-only secrets enabled. --> | |
| <div align="center"> | |
| <img src="https://capsule-render.vercel.app/api?type=rect&color=0:21154F,48:6045D8,100:19A98E&height=190§ion=header&text=DataPilot%20AI&fontSize=48&fontColor=ffffff&animation=fadeIn&desc=Autonomous%20Data%20Science%20%C2%B7%20Evidence-Grounded%20ML%20%C2%B7%20Explainable%20Decisions&descSize=17&descAlignY=72" width="100%" alt="DataPilot AI banner"> | |
| <p> | |
| <img src="https://readme-typing-svg.herokuapp.com/?font=Fira+Code&size=16&duration=2600&pause=850&color=6D4AFF¢er=true&vCenter=true&width=780&lines=CSV+or+Parquet+%E2%86%92+Quality+Audit+%E2%86%92+Validated+Model;8+Specialized+Agents+%7C+Stateful+LangGraph+Orchestration;Leakage-Safe+Pipelines+%7C+Critic+Retry+Loop+%7C+SHAP-Ready;Streamlit+%7C+FastAPI+%7C+DuckDB+%7C+scikit-learn" alt="DataPilot AI capabilities"> | |
| </p> | |
| [](https://www.python.org/) | |
| [](https://langchain-ai.github.io/langgraph/) | |
| [](https://datapilot-ai-agent.streamlit.app/) | |
| [](https://fastapi.tiangolo.com/) | |
| [](https://github.com/dineshbarri/DataPilot-AI/actions/workflows/ci.yml) | |
| [](LICENSE) | |
| **[API Docs](#fastapi)** · | |
| **[Architecture](docs/ARCHITECTURE.md)** · | |
| **[Security](docs/SECURITY.md)** · | |
| **[Model Governance](docs/MODEL_GOVERNANCE.md)** · | |
| **[Benchmarks](docs/BENCHMARKS.md)** · | |
| **[API Examples](docs/API_EXAMPLES.md)** | |
| </div> | |
| --- | |
| ## Why DataPilot AI? | |
| Most “AI data scientist” demos upload a CSV, run preprocessing on the entire dataset, compare | |
| a few models, and ask an LLM to write an impressive-sounding summary. That is fast—but it can | |
| leak test information, exaggerate confidence, and produce insights with no numerical provenance. | |
| **DataPilot AI treats trust as a feature.** It coordinates a stateful agent team that audits the | |
| data, plans the experiment, builds leakage-safe pipelines, compares models, challenges the | |
| winner, explains predictive signals, and exports reproducible artifacts. Every displayed metric | |
| comes from deterministic computation. The optional LLM may improve wording; it cannot create | |
| new numbers or execute code. | |
| ### Recruiter five-minute test | |
| 1. Open the app. | |
| 2. Keep **Iris classification** selected. | |
| 3. Click **Run autonomous analysis**. | |
| 4. Inspect the model comparison, critic decision, feature importance and full agent trace. | |
| 5. Download the fitted pipeline, model card and standalone report. | |
| No account, upload or API key is required. | |
| --- | |
| ## Product capabilities | |
| | Stage | What DataPilot does | Evidence produced | | |
| |---|---|---| | |
| | Data intake | Accepts bounded CSV, TSV, Excel, JSON, Parquet or packaged demos | row/column limits and validated schema | | |
| | Data quality | Detects missingness, duplicates, target gaps, imbalance, outliers and leakage-like fields | evidence registry with IDs, source and method | | |
| | EDA | Uses DuckDB and pandas for compact profiles, cardinality and correlations | dataset profile and statistical summary | | |
| | Planning | Infers classification/regression, primary metric, validation strategy and risk controls | typed Pydantic analysis plan | | |
| | Feature engineering | Builds numeric and categorical transformers inside the model pipeline | transformation plan and fitted pipeline | | |
| | Modeling | Selects linear, forest, extra-trees and optional XGBoost models by training-only CV | CV mean/stability and one-time untouched test results | | |
| | Evaluation | Applies thresholds and validation-consistency checks | critic approval, rejection reasons and retry count | | |
| | Explainability | Uses SHAP only when the optional dependency and fitted estimator are compatible; otherwise permutation importance | ranked predictive signals and caveats | | |
| | Reporting | Produces a dashboard, evidence-backed narrative and portable artifacts | HTML report, model card, JSON, joblib pipeline | | |
| | Follow-up | Answers questions from persisted run evidence | bounded, non-hallucinatory responses | | |
| --- | |
| ## Real agentic orchestration | |
| The “agents” are typed LangGraph nodes combining deterministic Python computation with optional, | |
| evidence-bounded LLM narration. The critic controls a conditional edge: | |
| weak or unstable analysis returns to the modeling node before explanation is allowed. | |
| ```mermaid | |
| flowchart LR | |
| A["CSV / Parquet / Demo"] --> B["Data Quality Agent"] | |
| B --> C["EDA Agent"] | |
| C --> D["Statistical Agent"] | |
| D --> E["Planning Agent"] | |
| E --> F["Feature Engineering Agent"] | |
| F --> G["Modeling Agent"] | |
| G --> H{"Evaluation / Critic"} | |
| H -->|"Reject + retry"| G | |
| H -->|"Approve"| I["Explainability Agent"] | |
| I --> J["Executive Insights Agent"] | |
| J --> K["Report · Model Card · Pipeline · Evidence"] | |
| ``` | |
| The Streamlit **Agent trace** tab shows every completed node, its duration and decision. | |
| --- | |
| ## Leakage-safe ML design | |
| ```python | |
| pipeline = Pipeline( | |
| [ | |
| ("preprocessor", ColumnTransformer(...)), | |
| ("model", candidate_model), | |
| ] | |
| ) | |
| # Imputers, encoders and scalers learn only from training folds. | |
| pipeline.fit(x_train, y_train) | |
| ``` | |
| - Split occurs before learned preprocessing. | |
| - Cross-validation refits the complete pipeline in every fold. | |
| - Classification defaults to balanced accuracy and stratification where possible. | |
| - Candidates are selected only by training-partition CV; the selected model touches the test set once. | |
| - Target-like names and identifier cardinality are flagged for human review. | |
| - Predictive importance is never described as causality. | |
| --- | |
| ## Dashboard experience | |
| The Streamlit application is summary-first and useful before any upload: | |
| - **Built-in demos:** Iris, Breast Cancer and Diabetes Progression. | |
| - **Executive overview:** findings, recommendations and quality risks. | |
| - **Model laboratory:** candidate comparison, CV stability and critic gate. | |
| - **Explainability:** interactive Plotly feature-importance view. | |
| - **Agent trace:** visible orchestration and retry behavior. | |
| - **Artifacts and Q&A:** downloads, evidence registry and run-specific questions. | |
|  | |
| The public demo URL is intentionally not claimed until a monitored deployment exists. Add a | |
| real browser capture under `assets/` together with the deployment URL after release validation. | |
| --- | |
| ## Technology stack | |
| | Layer | Technology | | |
| |---|---| | |
| | Agent orchestration | LangGraph typed state and conditional routing | | |
| | User interface | Streamlit and Plotly | | |
| | API | FastAPI and Pydantic | | |
| | Analytical engine | DuckDB, pandas and NumPy | | |
| | ML | scikit-learn; optional XGBoost and Optuna | | |
| | Explainability | optional SHAP with permutation fallback | | |
| | Persistence | SQLAlchemy; SQLite locally, PostgreSQL in production | | |
| | Experiment tracking | structured agent trace; optional MLflow/OpenTelemetry | | |
| | Artifacts | local storage interface, ready for S3/MinIO replacement | | |
| | Delivery | Docker Compose, Render Blueprint and Streamlit Cloud | | |
| | Quality | Pytest, Ruff, coverage, compile checks and Docker builds in GitHub Actions | | |
| --- | |
| ## Repository structure | |
| ```text | |
| DataPilot-AI/ | |
| ├── datapilot/ | |
| │ ├── config.py # typed environment configuration and limits | |
| │ ├── data.py # safe readers, DuckDB overview and sample datasets | |
| │ ├── quality.py # quality, leakage, imbalance, outlier and drift checks | |
| │ ├── modeling.py # leakage-safe pipelines and model comparison | |
| │ ├── workflow.py # LangGraph agent graph and critic loop | |
| │ ├── insights.py # deterministic + optional evidence-bounded narrative | |
| │ ├── reports.py # HTML report, model card, pipeline and JSON export | |
| │ ├── persistence.py # SQL run store and artifact interface | |
| │ ├── observability.py # optional MLflow integration | |
| │ └── safety.py # expression-only AST security policy | |
| ├── api/main.py # versioned FastAPI application | |
| ├── worker/main.py # optional isolated calculation worker | |
| ├── tests/ # quality, safety, API and end-to-end workflow tests | |
| ├── docs/ # architecture, deployment, security and governance | |
| ├── app.py # canonical premium Streamlit implementation | |
| ├── streamlit_app.py # Streamlit Cloud compatibility shim | |
| ├── Dockerfile* # non-root UI, API and worker images | |
| ├── docker-compose.yml # constrained local multi-service stack | |
| ├── render.yaml # Render API deployment blueprint | |
| └── .github/workflows/ci.yml # lint, tests, coverage, compile and image builds | |
| ``` | |
| --- | |
| ## Quick start | |
| ### Local Streamlit demo | |
| ```bash | |
| git clone https://github.com/dineshbarri/DataPilot-AI.git | |
| cd DataPilot-AI | |
| python -m venv .venv | |
| # Windows | |
| .venv\Scripts\activate | |
| # macOS/Linux | |
| source .venv/bin/activate | |
| pip install -e ".[dev]" | |
| streamlit run app.py | |
| ``` | |
| For the validated Python 3.12 reference environment, use `pip install -r requirements.lock`. | |
| The lock snapshot is refreshed after dependency updates and tested across supported interpreters. | |
| Open <http://localhost:8501>. | |
| ### Full AI/AutoML/observability extras | |
| ```bash | |
| pip install -e ".[all,dev]" | |
| ``` | |
| The default installation intentionally stays deployable on modest public-demo infrastructure. | |
| ### FastAPI | |
| ```bash | |
| uvicorn api.main:app --reload --port 8000 | |
| ``` | |
| Open <http://localhost:8000/docs>. | |
| Example: | |
| ```bash | |
| curl -X POST http://localhost:8000/v1/analyze/sample \ | |
| -H "Content-Type: application/json" \ | |
| -d "{\"sample\":\"iris\"}" | |
| ``` | |
| The API returns `202 Accepted` with a job ID. Poll `GET /v1/jobs/{job_id}` and cancel queued | |
| work with `DELETE /v1/jobs/{job_id}`. Set `API_KEY` in deployed environments and send it as | |
| `X-API-Key`. The bundled in-process queue is for development; use the production topology in | |
| `docs/ARCHITECTURE.md` for durable execution. | |
| ### Docker | |
| ```bash | |
| docker compose up --build | |
| ``` | |
| The optional worker runs with no network, a read-only filesystem, dropped capabilities, | |
| memory/CPU/PID limits and expression-only AST validation. | |
| --- | |
| ## Configuration | |
| Copy `.env.example` to `.env`. | |
| | Variable | Default | Purpose | | |
| |---|---|---| | |
| | `DATABASE_URL` | SQLite | Set a PostgreSQL URL for persistent production runs | | |
| | `ARTIFACT_ROOT` | `artifacts` | Root for reports, pipelines and model cards | | |
| | `MAX_UPLOAD_MB` | `25` | Public upload protection | | |
| | `MAX_ROWS` | `100000` | Maximum rows per analysis | | |
| | `MAX_COLUMNS` | `250` | Maximum feature width | | |
| | `MAX_CATEGORIES_PER_FEATURE` | `100` | Bound categorical expansion | | |
| | `MAX_ENCODED_FEATURES` | `5000` | Refuse unsafe estimated encoded width | | |
| | `API_KEY` | empty | Optional API authentication; required for public deployment | | |
| | `REQUESTS_PER_MINUTE` | `30` | Per-client API rate limit | | |
| | `MAX_CRITIC_RETRIES` | `1` | Conditional modeling retry budget | | |
| | `ENABLE_MLFLOW` | `false` | Enable optional experiment tracking | | |
| | `GEMINI_API_KEY` | empty | Optional narrative refinement only | | |
| No credential is embedded in the repository. | |
| --- | |
| ## Testing and engineering quality | |
| ```bash | |
| pip install -e ".[dev]" | |
| ruff check . | |
| pytest --cov=datapilot --cov=api --cov-fail-under=75 | |
| python -m compileall datapilot api worker app.py streamlit_app.py | |
| ``` | |
| CI runs Python 3.11, 3.12 and 3.13, enforces coverage, builds the package and all three images, | |
| audits dependencies, scans containers with Trivy, and performs secret detection. | |
| --- | |
| ## Deployment | |
| ### Streamlit Community Cloud | |
| - Entrypoint: `streamlit_app.py` | |
| - Python: 3.12 | |
| - Secrets: none required; `GEMINI_API_KEY` is optional | |
| - Default recruiter path: bundled demo dataset | |
| ### Render / Railway / Fly.io | |
| Deploy `Dockerfile.api`, attach PostgreSQL, and configure durable object storage if artifacts | |
| must survive container replacement. See [the deployment guide](docs/DEPLOYMENT.md). | |
| ### Production recommendation | |
| The included in-process job manager makes local requests non-blocking. For durable production | |
| jobs, replace it with a queue and separate workers, PostgreSQL state, S3-compatible artifacts, | |
| and short-lived sandboxed workers for any future code-execution capability. | |
| --- | |
| ## Responsible-use boundaries | |
| DataPilot is an exploratory decision-support system, not an automatic production approval | |
| authority. Before consequential use, complete: | |
| - target and leakage review | |
| - out-of-time and segment evaluation | |
| - privacy and retention assessment | |
| - fairness and disparate-impact evaluation | |
| - domain and legal approval | |
| - monitoring, rollback and retraining ownership | |
| See [Model Governance](docs/MODEL_GOVERNANCE.md) and [Security](docs/SECURITY.md). | |
| --- | |
| ## Roadmap | |
| - [ ] Background job queue and live progress streaming | |
| - [ ] S3/MinIO artifact adapter with signed downloads | |
| - [ ] Native PostgreSQL checkpoints for resumable LangGraph runs | |
| - [ ] Optuna study dashboard and experiment comparison | |
| - [ ] Time-series and clustering task families | |
| - [ ] Fairness and segment-performance report | |
| - [ ] Data-contract and schema-drift registry | |
| - [ ] Authenticated multi-tenant workspace | |
| --- | |
| ## Creator | |
| ### Dinesh Barri | |
| AI Engineer building agentic systems, data products, RAG applications and production-oriented | |
| machine-learning workflows. | |
| [](https://github.com/dineshbarri) | |
| [](https://www.linkedin.com/in/dinesh-barri-7654b010b) | |
| --- | |
| ## License | |
| Released under the [MIT License](LICENSE). | |
| If this project helps you, please star the repository and share the live demo. | |