Spaces:

SivaSai8143
/

auto-dev-agent

Sleeping

File size: 24,623 Bytes

---
license: mit
title: Auto Dev Agent
sdk: gradio
emoji: 💻
colorFrom: green
colorTo: indigo
---
# AutoDevAgent

> Autonomous code generation, execution, debugging, and testing — powered by LangGraph + Groq

[![HuggingFace Space](https://img.shields.io/badge/🤗%20HuggingFace-Live%20Demo-FFD21E)](https://huggingface.co/spaces/SivaSai8143/auto-dev-agent)
[![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python&logoColor=white)](https://www.python.org/)
[![License](https://img.shields.io/badge/License-MIT-green)](LICENSE)
[![LangSmith](https://img.shields.io/badge/LangSmith-Tracing-1C3C3C?logo=langchain&logoColor=white)](https://smith.langchain.com)
[![W&B](https://img.shields.io/badge/Weights_%26_Biases-Tracking-FFBE00?logo=weightsandbiases&logoColor=000)](https://wandb.ai)

---

## What is this?

AutoDevAgent is a multi-agent AI system that takes a natural language task description and autonomously writes, runs, debugs, and tests Python or SQL code — without any manual intervention. It uses a LangGraph state machine to coordinate nine specialised agents, each responsible for one step of the software development loop: planning, generation, execution, error classification, debugging, test generation, explanation, and flowchart diagramming. The system is designed for developers who want to see how agentic AI handles real code quality requirements — not just code generation, but full-cycle verification including automated unit tests and self-healing retry loops.

---

## Live Demo

🤗 [Try it on HuggingFace Spaces](https://huggingface.co/spaces/SivaSai8143/auto-dev-agent)

<video src="https://huggingface.co/spaces/SivaSai8143/auto-dev-agent/resolve/main/assets/auto-dev-agent-demo.mp4" controls autoplay loop muted width="100%"></video>

---

## Tech Stack

| Component           | Tool                  | Purpose                                                              |
|---------------------|-----------------------|----------------------------------------------------------------------|
| Agent Orchestration | LangGraph 0.2+        | State machine with conditional edges, retry loops, and streaming     |
| LLM Framework       | LangChain 0.3+        | LLM client abstraction, message formatting, prompt templates         |
| LLM Provider        | Groq API              | Fast inference for Llama 3.1 8B, Llama 4 Scout 17B, Llama 3.3 70B   |
| State Management    | Pydantic V2           | Typed, validated, immutable pipeline state shared across all agents  |
| UI                  | Gradio 6.0+           | Streaming reactive UI with live pipeline visualiser                  |
| Python Execution    | subprocess (stdlib)   | Isolated Python code execution with 15-second timeout                |
| SQL Execution       | sqlite3 (stdlib)      | In-memory SQLite for SQL query execution and test assertions          |
| Observability       | LangSmith             | Per-call LLM tracing, latency, token counts, and prompt inspection   |
| Experiment Tracking | Weights & Biases      | Benchmark run logging, success rate trends, token usage metrics      |
| Deployment          | HuggingFace Spaces    | Zero-config hosting with Gradio auto-detection                       |

---

## Total Cost

| Item | Cost |
|---|---|
| HuggingFace Spaces hosting | Free |
| Groq API — Llama 3.1 8B (classification, flowchart) | Free (14,400 req/day) |
| Groq API — Llama 4 Scout 17B (planning, tests, explanation) | Free (100 emails/day) |
| Groq API — Llama 3.3 70B (code generation, debugging) | Free (1,000 req/day) |
| LangSmith — LLM tracing | Free (5,000 traces/month) |
| Weights & Biases — experiment tracking | Free (unlimited for personal use) |
| **Total** | **0** |

---
## Architecture

AutoDevAgent runs every task through a fixed-topology LangGraph state machine. The user's task enters the pipeline at the top, passes through a sequence of specialised agents, and either exits as a successful result or loops back through a debug cycle. The pipeline has two independent retry budgets — one for execution failures (code crashes) and one for test failures (code runs but tests don't pass) — so neither type of failure escalates to human intervention prematurely.

### Agent Pipeline

```
User Task
    │
    ▼
[Clarification] ──── unclear ────► ask user, then restart
    │ clear
    ▼
[Language Detect] ──────────────── Python / SQL / override
    │                               (model routing also runs here,
    │                                before the LangGraph pipeline starts)
    ▼
[Planner] ──────────────────────── breaks task into labelled steps
    │
    ▼
[Code Generator] ───────────────── writes Python function or SQL query
    │
    ▼
[Executor] ─────────────────────── runs code in subprocess / SQLite
    │
    ├── PASS ──────────────────────────────────────────────┐
    │                                                       │
    └── FAIL ──► [Error Classifier] ──► [Debug Agent] ─── retry (max 5)
                                                │
                                        max retries hit?
                                                │
                                   ┌────────────┴────────────┐
                              execution fail             test fail
                                   │                         │
                            [HITL panel]             regen (max 2×)
                            user edits code          then PARTIAL_SUCCESS

    PASS ──► [Test Generator] ──► [Test Runner]
                                       │
                        ┌──────────────┴──────────────┐
                       PASS                          FAIL
                        │                              │
                   [Explanation]            [Debug Agent] ──► retry (max 3)
                   [Flowchart]                         │
                        │                    max test-fix cycles hit?
                   [SUCCESS]                           │
                                               [PARTIAL_SUCCESS]
```

### Project Structure

```
autodevagent/
│
├── app.py                          # Gradio UI entry point — wires all components
├── config.py                       # All settings: API keys, model names, retry limits
├── requirements.txt                # Pinned dependencies
│
├── agents/
│   ├── clarification_agent.py      # Checks if the task is specific enough to execute
│   ├── code_generator.py           # Writes Python functions or SQL queries
│   ├── debug_agent.py              # Diagnoses failures and rewrites code to fix them
│   ├── detect_agent.py             # Classifies task language + scope gate + coding gate
│   ├── explanation_agent.py        # Writes plain-English explanation of the final code
│   ├── flowchart_agent.py          # Generates Mermaid.js logic diagram
│   ├── model_router.py             # Assigns best-fit model to each agent role
│   ├── planning_agent.py           # Decomposes the task into labelled plan steps
│   ├── polisher_agent.py           # Optional safe refactors with diff display
│   ├── test_generator.py           # Writes unittest (Python) or assertion (SQL) tests
│   └── __init__.py
│
├── pipeline/
│   ├── graph.py                    # LangGraph StateGraph — all nodes, edges, routing
│   ├── state.py                    # Pydantic V2 PipelineState — single source of truth
│   ├── router.py                   # Conditional edge helpers
│   └── __init__.py
│
├── executors/
│   ├── python_executor.py          # Runs Python code in a subprocess with timeout
│   ├── sql_executor.py             # Runs SQL against in-memory SQLite
│   ├── test_runner.py              # Runs generated unittest file, parses results
│   └── __init__.py
│
├── ui/
│   ├── pipeline_visualiser.py      # Animated HTML/CSS pipeline card renderer
│   ├── components.py               # Reusable Gradio component builders
│   ├── session_history.py          # Per-session run history tracker
│   └── __init__.py
│
├── evaluation/
│   ├── benchmark.py                # 5 fixed benchmark tasks (3 Python + 2 SQL)
│   ├── metrics.py                  # Aggregates success rate, tokens, iterations
│   └── __init__.py
│
├── observability/
│   ├── langsmith_tracer.py         # LangSmith trace setup and token extraction
│   ├── wandb_tracker.py            # W&B benchmark run logger
│   └── __init__.py
│
└── utils/
    ├── dependency_manager.py       # Auto-installs missing pip packages at runtime
    ├── error_cache.py              # Tracks seen errors to force different fix strategies
    ├── token_counter.py            # Accumulates token usage across all LLM calls
    └── __init__.py
```

---

## Features

### Core Agent Loop

1. **Clarification gate** — before running anything, checks if the task is specific enough. If ambiguous, asks one targeted question and waits for the user's answer before proceeding.
2. **Non-coding gate** — rejects general knowledge questions with a friendly message. Only blocks on high/medium confidence to avoid false positives.
3. **Scope gate** — rejects full-project requests ("build a YouTube app"). Any single SQL query is always in-scope regardless of complexity.
4. **Language auto-detection** — fast 8B model classifies Python vs SQL. User can override via radio button at any time.
5. **Task-aware model routing** — analyses the task and assigns the best model to each agent role. Reasoning-heavy and SQL tasks route Llama 3.3 70B to the generator and debugger; planning, tests, and explanation always use Llama 4 Scout; error classification and flowchart use Llama 3.1 8B.
6. **Structured planning** — PlanningAgent decomposes the task into typed, labelled steps before any code is written.
7. **Code generation** — CodeGeneratorAgent writes a self-contained Python function or SQL query using the plan as a scaffold.
8. **Sandboxed execution** — Python code runs in a `subprocess` with a 15-second timeout; SQL runs in an in-memory SQLite instance with a 10-second timeout.
9. **Error classification** — fast 8B model classifies every failure as `syntax | runtime | logic | timeout` before the debug agent runs, so the fix strategy is targeted rather than generic.
10. **Self-healing debug loop** — DebugAgent rewrites the code up to 5 times per attempt. After 5 failures it attempts a full code regeneration (up to 2 regens). For execution failures that exhaust all retries, escalates to Human-in-the-Loop.
11. **Error cache** — tracks every error message seen in the current run. If the same error recurs, forces the debug agent to try a completely different fix strategy instead of repeating the same failing approach.
12. **Two-tier retry budgets** — test failures and execution failures have completely separate retry counters (`max_test_fix_retries=3`, `max_debug_retries=5`). Test failures never escalate to HITL — they exit as `PARTIAL_SUCCESS` after all retries.
13. **Human-in-the-Loop panel** — when execution failures exhaust all retry budgets, presents the user with the last error, a code editor, and fix-strategy options. Resubmission re-runs the full pipeline with the user's edits as context.

### SQL Support

- **Schema inference** — for SQL tasks with no schema given, the PlanningAgent infers a realistic table structure from the task description.
- **Dummy data generation** — the planner generates `INSERT` statements with representative data so the query can be executed and verified against known values.
- **SQLite execution** — all SQL runs against an in-memory SQLite database; no external database required.
- **Assertion-based tests** — SQL tests use positional result access (`row[0]`, `row[1]`) rather than column names, ensuring correctness even for aggregate expressions like `MAX(salary)` that SQLite returns as raw strings.

### Test Generation

After code executes successfully, TestGeneratorAgent writes a `unittest` test suite (Python) or a set of Python `sqlite3` assertion tests (SQL). Tests are frozen as a stable target — the same suite runs on every retry cycle so the debug agent has a consistent benchmark to aim for. Tests cover a happy-path case, an edge case, and a boundary case, each with a docstring explaining the exact input and expected outcome. Tests must use independently derived input values — not the same example values shown in the code's execution output.

### Auto-Polisher

PolisherAgent is implemented as a standalone module (`agents/polisher_agent.py`) that applies safe, cosmetic refactors — renaming unclear variables, adding docstrings, extracting repeated logic into helpers, removing redundant code — and returns a `difflib` unified diff of the changes. The agent is fully implemented but is not yet connected to the Gradio UI; it is ready to be wired in as a "Polish" button in a future release.

### Code Explanation & Flowchart

ExplanationAgent writes a plain-English walkthrough of how the final code works, targeted at someone who can read code but didn't write it. FlowchartAgent generates a Mermaid.js `graph TD` flowchart of the logic. Flowcharts are only generated for Python code longer than 20 lines (or always for SQL) — below that threshold, the diagram adds no value over reading the code directly. The Mermaid output goes through a four-layer sanitisation pipeline: prompt rules → LLM syntax validation → LLM fix retry → parse-and-rebuild in `app.py`. The rebuild step also drops orphaned nodes (nodes not reachable from the start node via BFS) automatically.

### Observability

- **LangSmith tracing** — when `enable_langsmith=True`, every LLM call is traced with prompt, response, latency, and token count. Key validation runs at startup to silently disable tracing if the API key is invalid.
- **Token counter** — `TokenUsage` accumulates prompt and completion tokens across all agents in a run. Displayed live in the stats strip after each run.
- **Rate limit warning** — monitors request count against Groq's free tier limit (30 RPM). Fires a warning at 80% usage.
- **Model switcher** — dropdown in the UI lets users pin a specific model or leave it on Auto for the model router to decide.
- **Error cache** — `ErrorCache` in `utils/error_cache.py` stores a hash of every error seen in the current run. Injected into the debug prompt to force novel fix attempts.

### Pipeline Visualiser

The right column of the UI shows a live animated pipeline card for each agent node. Cards are rendered as a complete HTML/CSS string on every LangGraph streaming yield — the current state is baked directly into CSS class names (`av-running`, `av-done`, `av-pending`, `av-error`) so no JavaScript is needed. Each card shows the agent name, its assigned model (from the model router), and its current status with a colour-coded glow animation.

### Evaluation Harness

A built-in benchmark runs 5 fixed tasks through the full pipeline:

| # | Task                       | Language |
|---|----------------------------|----------|
| 1 | Reverse a string           | Python   |
| 2 | Fibonacci sequence         | Python   |
| 3 | Find duplicates in a list  | Python   |
| 4 | Top N customers by revenue | SQL      |
| 5 | Department headcount query | SQL      |

Results are logged to W&B with success rate, average debug iterations, average tokens per task, and total execution time. The harness is accessible from the **Evaluation harness** accordion in the UI.

---

## Benchmark Results

All 5 tasks run end-to-end through the full pipeline — clarification → detect → plan → generate → execute → test → explain — with no manual intervention. Model: Llama 4 Scout 17B for all primary roles · Llama 3.1 8B for classification and flowchart.

| Task                     | Language | Status  | Iterations | Tokens | Time   |
|--------------------------|----------|---------|------------|--------|--------|
| Reverse string           | Python   | ✅ Pass | 0          | 3,390  | 3.5s   |
| Fibonacci sequence       | Python   | ✅ Pass | 0          | 3,899  | 4.2s   |
| Find duplicates          | Python   | ✅ Pass | 0          | 4,124  | 4.2s   |
| Top customers by revenue | SQL      | ✅ Pass | 0          | 3,173  | 3.7s   |
| Department headcount     | SQL      | ✅ Pass | 0          | 4,308  | 11.5s  |

**5/5 tasks passed · 0 debug iterations on every task · avg 3,779 tokens/task · avg 5.4s/task**

All tasks passed on the first generation attempt with zero debug iterations needed. Python tasks each ran 6 unit tests; SQL tasks ran 3 assertion tests. All 24 tests passed across the full run.

---

## Getting Started

### Prerequisites

- Python 3.10 or higher
- A [Groq API key](https://console.groq.com) — required
- A [LangSmith API key](https://smith.langchain.com) — optional, for LLM tracing
- A [Weights & Biases API key](https://wandb.ai) — optional, for benchmark tracking

### Installation

```bash
# 1. Clone the repo
git clone https://github.com/sivasaiyadav8143/autodevagent.git
cd autodevagent

# 2. Install dependencies
pip install -r requirements.txt

# 3. Set up environment variables
cp .env.example .env
# Edit .env and add your API keys

# 4. Run
python app.py
# Opens at http://localhost:7
```

### Environment Variables

| Variable            | Required | Where to get it                                                   |
|---------------------|----------|-------------------------------------------------------------------|
| `GROQ_API_KEY`      | ✅ Yes   | [console.groq.com](https://console.groq.com)                     |
| `LANGSMITH_API_KEY` | Optional | [smith.langchain.com](https://smith.langchain.com)               |
| `LANGCHAIN_PROJECT` | Optional | Any string — groups traces in LangSmith. Default: `autodevagent` |
| `WANDB_API_KEY`     | Optional | [wandb.ai/authorize](https://wandb.ai/authorize)                 |
| `WANDB_PROJECT`     | Optional | Any string — groups runs in W&B. Default: `autodevagent`         |

To enable LangSmith tracing, also set `enable_langsmith=True` in your `.env`.
To enable W&B tracking, also set `enable_wandb=True` in your `.env`.

---

## Usage

### Running a Python Task

1. Type a task in the **Task description** box, e.g. *"Write a function that checks if two strings are anagrams, case-insensitive, ignoring spaces and punctuation"*
2. Leave **Language** on `Auto (recommended)` — the detect agent will pick Python.
3. Leave **Model** on `Auto` — the model router will assign the best model per role.
4. Click **▶ Run**.
5. Watch the pipeline visualiser animate each agent as it fires.
6. Results appear in the **Code**, **Tests**, **Explanation**, and **Flowchart** tabs.

### Running a SQL Task

1. Type a SQL task, e.g. *"Write a query to find the second highest salary from an employees table. Assume no schema is given — infer it."*
2. Language will be auto-detected as SQL. The planner will infer an `employees` schema and generate `INSERT` dummy data.
3. The SQL executor runs the query against an in-memory SQLite database.
4. The test generator writes three assertion tests using positional row access.

### Running the Benchmark

1. Scroll to the **Evaluation harness** accordion at the bottom of the left column.
2. Select a model from the dropdown (or leave on Auto).
3. Click **▶ Run benchmark**.
4. Results are displayed as a Markdown table and logged to W&B if enabled.

---

## Known Limitations

1. **SQL test data is inferred, not real** — the planner generates dummy `INSERT` data based on the task description. For complex business queries, the dummy data may not cover all edge cases, causing tests to pass on simplified data but fail on real data.
2. **subprocess isolation, not Docker** — Python code runs in a subprocess on the host machine. There is no container sandbox, so code has filesystem and network access. Do not run untrusted code.
3. **No C-extension support** — packages that require C compilation (e.g. `numpy`, `pandas` with native binaries) may not install correctly in the subprocess environment on all platforms.
4. **Mermaid diagram quality degrades for complex logic** — deeply nested conditionals, multiple recursive branches, and long functions produce simplified or inaccurate flowcharts. The diagram describes logical flow, not exact control flow.
5. **Groq free tier rate limits** — the free tier allows 30 requests per minute. A single pipeline run with multiple debug retries can use 10–15 requests. The rate limit warning fires at 80% usage; heavy benchmarking may hit the limit.

---

## Future Improvements

1. **Docker sandbox** — run generated code inside a Docker container for true isolation and consistent package environments.
2. **SQL stress test mode** — generate multiple sets of dummy data with edge cases (nulls, empty tables, duplicate keys) and run the query against all of them.
3. **JavaScript and Bash support (V2)** — extend the pipeline to support JS (Node.js subprocess) and Bash scripts.
4. **Full database support** — connect to PostgreSQL or MySQL instead of SQLite, allowing real schema and data to be used for SQL tasks.
5. **Persistent session storage** — save session history to disk so completed runs survive page refreshes.
6. **CI/CD for automated benchmarks** — run the benchmark harness on every commit and post results to W&B automatically.

---

## Design Decisions

**Why LangGraph over a manual loop**

The debug loop — generate → execute → fail → classify → fix → repeat — is a cycle with conditional exits. Implementing this as a manual `while` loop with if-else branches works but becomes unmaintainable as retry budgets, regen counters, and test-fix cycles are added. LangGraph expresses the same logic as a state machine with named nodes and typed edges. Each node is a pure function (receives state, returns partial update), making the flow readable, debuggable, and easy to extend with new agents.

**Why Pydantic V2 for pipeline state**

Every agent in the pipeline reads from and writes to a shared `PipelineState` object. Using Pydantic V2 means every field is typed and validated — a bug where an agent sets `debug_iterations` to a string instead of an int is caught immediately at assignment time, not three agents later when something tries to compare it. The `model_copy()` method makes immutable partial updates clean and safe.

**Why the error cache forces a different fix strategy**

Without the error cache, the debug agent will sometimes generate the same broken code twice in a row — especially for logic errors where the first "fix" attempts the same approach. The cache stores a hash of every error message seen in the current run and injects the full history into the debug prompt: *"You have already tried X and Y — both produced the same error. Try a completely different approach."* This breaks the loop and forces genuine variation in fix strategies.

**Why the flowchart has a 20-line threshold**

For short functions (under 20 lines), a Mermaid flowchart adds little beyond what reading the code directly provides. The diagram overhead — one LLM call, post-processing, rendering — is only worth it when the function is long enough that the logical flow is non-obvious. SQL queries always get a flowchart because even a short query (`SELECT MAX(salary) FROM employees`) has a meaningful multi-step logical flow (load table → find maximum → return result) that benefits from visualisation.

---

## License

MIT — see [LICENSE](LICENSE) for details.

---

## Author

**Siva Sai Yadav**

[HuggingFace](https://huggingface.co/SivaSai8143) · [LinkedIn](https://linkedin.com/in/sivasai-mudugandla/)

---

*Built as an open-source portfolio project using entirely free infrastructure.*