Spaces:
Sleeping
Sleeping
File size: 14,509 Bytes
3ae68d6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 | # Project Master Plan: Smart API DevTool
This document serves as the master coordinator for the **Smart API DevTool** project. It outlines our architectural roadmap, directory structure, Git workflow, and connects all local configuration, lesson, and implementation tracking files.
---
## 1. Project Overview & Architecture
This tool is designed to demonstrate python-based backend engineering, docker configurations, and modern agentic architectures:
1. **Interactive Web Dashboard**: A local Python FastAPI server hosting a premium, dark-themed responsive single-page HTML/CSS/JS frontend.
2. **Model Context Protocol (MCP) Server**: A standard protocol wrapper enabling AI agents (like Claude Desktop or Cursor) to call your scraping and wrapper-generation engine natively from the IDE.
### System Architecture Flow (Unified Core Engine)
Our Core Engine uses **LangGraph** to orchestrate a single-agent **Self-Healing Test Loop** that runs generated unit tests inside our Docker container, captures failures, and auto-corrects code errors.
```
┌────────────────────────────────┐
│ Developer Request / Input │
└───────────────┬────────────────┘
│ (URL or Paste Docs)
▼
┌────────────────────────────────┐
│ Core Scraping Service │
│ (Firecrawl Keyless / Local) │
└───────────────┬────────────────┘
│ (Clean Markdown Docs)
▼
┌────────────────────────────────┐
│ Pre-Generation Grounding Check │
│ (Validate REST API Specs) │
└───────┬────────────────┬───────┘
│ │
(Specs Found) ▼ ▼ (No Specs Found)
┌────────────────┐ ┌────────────────┐
│ Proceed to Gen │ │ Raise error │
└───────┬────────┘ │ (Fail Fast UI) │
│ └────────────────┘
▼
┌────────────────────────────────┐
│ Unified Provider Router │
│ (Gemini, Groq, OpenRouter, │
│ or Ollama) │
└───────┬────────────────┬───────┘
│ │
(Cloud API) ▼ ▼ (Local: qwen2.5-coder)
┌──────────────┐ ┌──────────────┐
│ Cloud API │ │ Ollama API │
│ (Gemini/Groq/│ └──────┬───────┘
│ OpenRouter) │ │
└──────┬───────┘ │
│ │
└────────┬────────┘
│ (Initial Wrapper & Test Suite Code)
▼
┌────────────────────────────────┐
│ LangGraph: Generate Node │◄──────────┐
│ (State: code, tests, errors) │ │
└───────────────┬────────────────┘ │ (Failed)
│ │ Route back
▼ │ with error
┌────────────────────────────────┐ │ logs
│ LangGraph: Test Node │ │
│ (Executes tests inside the │ │
│ Docker container using local │ │
│ subprocesses: pytest, node, │ │
│ ts-node, go, java) │ │
└───────────────┬────────────────┘ │
│ │
├─► [Test Fails & Retries < 3]─┘
│
▼ [Test Passes OR Retries >= 3]
┌────────────────────────────────┐
│ END / Deliver │
└───────┬────────────────┬───────┘
│ │
▼ ▼
┌────────────────────────┐┌────────────────────────┐
│ Web UI Output Panel ││ MCP Client Response │
│ (Browser Download) ││ (IDE Code Write) │
└────────────────────────┘└────────────────────────┘
```
---
## 2. Directory Structure
```
smart-api-devtool/
├── requirements.txt - Python dependencies (fastapi, uvicorn, langgraph, google-genai)
├── main.py - Entrypoint: launches FastAPI server or MCP server mode
├── Dockerfile - Docker setup pre-installing Python, Node, Go, and Java runtimes
├── ProjectPlan.md - Master coordination plan (This file)
├── MISSION.md - Project goals and learning mission
├── RESOURCES.md - Curated list of documentation and wisdom links
├── NOTES.md - Local notes & preferences
├── OutputFormat.md - Layout formats for code outputs
├── AboutMe.md - Author profile and hackathon directives
├── PRD.md - Product Requirements Document
├── src/
│ ├── config.py - Application configurations & env validation
│ ├── app.py - FastAPI server routes, static assets serving, and CORS config
│ ├── mcp_server.py - Model Context Protocol integration handler
│ ├── agent.py - LangGraph Agent Definition (State, Nodes, validation, and Graph)
│ └── services/
│ ├── scraper.py - Firecrawl HTTP client (supports keyless & API keys)
│ └── executor.py - Subprocess code runner executing python/node/go/java sandboxes
├── public/ - Web Dashboard Assets
│ ├── index.html - High-end dark theme dashboard
│ ├── style.css - Custom glassmorphic styles & animations
│ └── app.js - Frontend application controller
├── lessons/ - Curriculum for developer topics
└── learning-records/ - Log of key decisions and insights
```
---
## 3. Project References
Use these clickable file links to inspect details, check tasks, and verify progress:
### Core Configuration Files
* **[MISSION.md](file:///d:/Downloads/Projects/My%20College%20Projects/Smart%20DevTool%20for%20API%20Integration/MISSION.md)**: Goals and scope boundaries.
* **[RESOURCES.md](file:///d:/Downloads/Projects/My%20College%20Projects/Smart%20DevTool%20for%20API%20Integration/RESOURCES.md)**: Curated scraping and API development guides.
* **[NOTES.md](file:///d:/Downloads/Projects/My%20College%20Projects/Smart%20DevTool%20for%20API%20Integration/NOTES.md)**: Scratchpad of active work items and preferences.
* **[Output Format Guide](file:///d:/Downloads/Projects/My%20College%20Projects/Smart%20DevTool%20for%20API%20Integration/OutputFormat.md)**: Details on ZIP structures and generated code client classes.
* **[Developer Profile](file:///d:/Downloads/Projects/My%20College%20Projects/Smart%20DevTool%20for%20API%20Integration/AboutMe.md)**: Bio and hackathon placement goals.
* **[PRD Specifications](file:///d:/Downloads/Projects/My%20College%20Projects/Smart%20DevTool%20for%20API%20Integration/PRD.md)**: Product Requirements Document with user stories and decisions.
### System-Generated Artifacts
* **[Implementation Plan](file:///C:/Users/yashw/.gemini/antigravity/brain/bfcca984-987f-419e-96de-3d92c17e1877/implementation_plan.md)**: Detailed technical specifications for backend services, API structures, and frontend routes.
* **[Active Task Checklist](file:///C:/Users/yashw/.gemini/antigravity/brain/bfcca984-987f-419e-96de-3d92c17e1877/task.md)**: Current completion status of specific development steps.
* **[Walkthrough Report](file:///C:/Users/yashw/.gemini/antigravity/brain/bfcca984-987f-419e-96de-3d92c17e1877/walkthrough.md)**: Summary of final work, validations, and testing outputs.
---
## 4. Phase-by-Phase Roadmap
### **Phase 1: Project Setup & Core Services (Completed)**
* Configure Python virtual environment and `requirements.txt`.
* Write the `Dockerfile` installing Python, Node, Go, and Java runtimes.
* Set up FastAPI app boilerplate (`main.py`, `src/app.py`).
* Code `scraper.py` using Firecrawl REST interface.
* Code test runner `executor.py` utilizing Python subprocesses to execute language-specific test runs inside a `/temp` folder sandbox.
* *Verification*: Verify URL scraping and local test-suite executions via scripts.
### **Phase 2: LangGraph Self-Healing Agent (Completed)**
* Define the LangGraph State, `generate` Node, and `test` Node.
* Code the conditional transition edge (Self-Heal Loop up to 3 retries).
* Connect the compiled LangGraph to the FastAPI endpoints.
* *Verification*: Verify self-healing loop by running local tests.
### **Phase 3: MCP Server Integration (Completed)**
* Implement `mcp_server.py` using standard Model Context Protocol stream handlers.
* Bind standard CLI flag (`python main.py --mcp`) to run the MCP server.
### **Phase 4: Premium Web UI & Exporter (Completed)**
* Design HTML5 layout with glassmorphism CSS aesthetics.
* Incorporate model selector dropdowns (Gemini, Ollama, Groq, OpenRouter), and live terminal logging showing the self-healing test runs.
* Expose code exports (downloading wrapper, unit tests, and README file in a single ZIP).
### **Phase 5: Hugging Face Spaces Deployment (Completed)**
* Deploy the application to a Hugging Face Space using the custom Dockerfile.
* *Verification*: Test the hosted web service dynamically in the cloud.
### **Phase 6: Grounding Hardening & Git Delivery (Completed)**
* Implement LLM-based pre-generation documentation checks to prevent endpoint hallucinations.
* Apply strict dependency-free test execution rules across all five supported languages.
* Format the codebase, compile the walkthrough report, and commit changes using Conventional Commit patterns.
---
## 5. Future System Design Proposals
### Architectural Question: Concurrency & Sandbox Isolation on Free Cloud Runtimes
> *"If there are 10 to 100 users using a local host, when they are using local host and if they do not have the package installed, we do not want to disturb their local system. If we use the Docker availability in Hugging Face and make it a common platform to run all the tasks for those 100 users, there is a resources bottleneck on the free tier (2 vCPUs, 16 GB RAM). How can we resolve this on a free-tier basis to improve project system design?"*
### Proposed Solution: Piston-Based Serverless Code Execution
To eliminate compiler installation overhead inside the host Docker image and prevent CPU/RAM resource starvation under concurrent user spikes, we propose offloading sandbox executions to a serverless code execution engine:
#### 1. How it Works
- Instead of executing tests locally via Python subprocesses (`pytest`, `node`, `go test`, `javac`), the sandbox executor (`src/services/executor.py`) delegates execution to a **Piston API** endpoint.
- The executor constructs an HTTP POST request containing the generated wrapper source code, the test file code, and the language identifier, sending it to a public or private Piston API instance:
```http
POST /api/v2/execute HTTP/1.1
Host: emkc.org
Content-Type: application/json
{
"language": "python",
"version": "3.10.0",
"files": [
{
"name": "client.py",
"content": "class MyAPIClient: ..."
},
{
"name": "test_client.py",
"content": "def test_get_user(): ..."
}
]
}
```
- The Piston container runs the tests in isolation and returns structured JSON containing the stdout, stderr, and process exit code.
#### 2. Architectural Impact
* **Pros:**
- **Ultra-Lightweight Containers:** Deletes heavy compilers (golang, default-jdk, nodejs) from the custom Dockerfile, shrinking the image size from ~2GB to <200MB.
- **Zero CPU/RAM Contention:** Shifting the compilation work off the Hugging Face Space protects it from OOM (Out of Memory) crashes when 100 concurrent users trigger code executions.
- **No Local Installs:** Users do not need any local compiler packages on their system, maintaining complete environment isolation.
* **Cons:**
- Introduces a network dependency on third-party public execution endpoints, which may be subject to external rate limits or availability.
|