Spaces:
Sleeping
Sleeping
| title: LLM Code Deployment API | |
| emoji: 🚀 | |
| colorFrom: blue | |
| colorTo: green | |
| sdk: docker | |
| app_port: 7860 | |
| # LLM Code Deployment System | |
| An automated system for building, deploying, and evaluating LLM-generated web applications with GitHub Pages integration. | |
| ## Overview | |
| This project implements a complete workflow for: | |
| - **Students**: Receive task requests, use LLMs to generate code, deploy to GitHub Pages, and submit for evaluation | |
| - **Instructors**: Generate task requests, receive submissions, run automated evaluations (static, dynamic, and LLM-based) | |
| ## 🚀 Quick Deployment for Students | |
| **Deploy to Hugging Face Spaces in 10 minutes:** | |
| 1. Read **[DEPLOYMENT.md](DEPLOYMENT.md)** for complete step-by-step instructions | |
| 2. Read **[README_SPACES.md](README_SPACES.md)** for Hugging Face Spaces configuration | |
| 3. Get your AIPipe token from https://aipipe.org/login ($2/month free for IIT Madras students) | |
| 4. Create a Space at https://huggingface.co/new-space | |
| 5. Configure environment variables and deploy! | |
| **Already deployed?** Just submit your endpoint URL to the instructor's Google Form! | |
| ## Architecture | |
| ``` | |
| ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ | |
| │ Instructor │ │ Student │ │ GitHub │ | |
| │ System │────────▶│ API │────────▶│ Pages │ | |
| │ │ POST │ │ Deploy │ │ | |
| └─────────────┘ Task └──────────────┘ └─────────────┘ | |
| │ │ │ | |
| │ │ │ | |
| │ └────────POST─────────────┘ | |
| │ Submission │ | |
| │ │ | |
| ▼ ▼ | |
| ┌─────────────┐ ┌─────────────┐ | |
| │ Evaluation │◀──────────────────────────────────│ Validation │ | |
| │ Database │ │ & Checks │ | |
| └─────────────┘ └─────────────┘ | |
| ``` | |
| ## Features | |
| ### Student-Side | |
| - **API Endpoint**: Receives task requests via HTTP POST | |
| - **LLM Code Generation**: Uses Claude/GPT to generate complete web apps | |
| - **GitHub Integration**: Automatically creates repos, pushes code, enables Pages | |
| - **Automatic Notification**: Sends repo details to evaluation endpoint | |
| - **Round 2 Support**: Handles update requests for existing repos | |
| ### Instructor-Side | |
| - **Task Templates**: YAML-based parametrizable task definitions | |
| - **Round 1 & 2 Scripts**: Automated task generation and distribution | |
| - **Evaluation API**: Receives and validates student submissions | |
| - **Multi-Level Checks**: | |
| - Static: License, README, repo creation time, secrets detection | |
| - LLM: Code quality, documentation quality | |
| - Dynamic: Playwright-based functional testing | |
| - **Database**: PostgreSQL storage for tasks, repos, and results | |
| ## Project Structure | |
| ``` | |
| tds-p1/ | |
| ├── shared/ # Shared utilities and models | |
| │ ├── config.py # Configuration management | |
| │ ├── models.py # Pydantic data models | |
| │ ├── logger.py # Logging setup | |
| │ └── utils.py # Utility functions | |
| ├── student/ # Student-side components | |
| │ ├── api.py # FastAPI endpoint | |
| │ ├── code_generator.py # LLM-based code generation | |
| │ ├── github_manager.py # GitHub operations | |
| │ └── notification_client.py # Evaluation notification | |
| ├── instructor/ # Instructor-side components | |
| │ ├── api.py # Evaluation endpoint | |
| │ ├── database.py # Database models and operations | |
| │ ├── task_templates.py # Template management | |
| │ ├── round1.py # Round 1 task generation | |
| │ ├── round2.py # Round 2 task generation | |
| │ ├── evaluate.py # Main evaluation script | |
| │ └── checks/ # Evaluation modules | |
| │ ├── static_checks.py # Static analysis | |
| │ ├── dynamic_checks.py # Playwright tests | |
| │ └── llm_checks.py # LLM evaluations | |
| ├── templates/ # Task template YAML files | |
| │ ├── sum-of-sales.yaml | |
| │ ├── markdown-to-html.yaml | |
| │ └── github-user-created.yaml | |
| ├── pyproject.toml # Project dependencies | |
| ├── .env.example # Environment variables template | |
| └── README.md # This file | |
| ``` | |
| ## Setup | |
| ### Prerequisites | |
| - Python 3.10+ | |
| - PostgreSQL database | |
| - GitHub account with personal access token | |
| - Anthropic or OpenAI API key | |
| ### Installation | |
| 1. **Clone the repository** | |
| ```bash | |
| git clone <your-repo-url> | |
| cd tds-p1 | |
| ``` | |
| 2. **Install dependencies** | |
| ```bash | |
| pip install -e . | |
| ``` | |
| 3. **Install Playwright browsers** | |
| ```bash | |
| playwright install chromium | |
| ``` | |
| 4. **Configure environment** | |
| ```bash | |
| cp .env.example .env | |
| # Edit .env with your credentials | |
| ``` | |
| 5. **Set up database** | |
| ```bash | |
| # Create PostgreSQL database | |
| createdb llm_deployment | |
| # Initialize tables | |
| python -c "from instructor.database import Database; Database().create_tables()" | |
| ``` | |
| ## Configuration | |
| Edit `.env` with your settings: | |
| ### Student Configuration | |
| ```bash | |
| STUDENT_SECRET=your-secret-key | |
| STUDENT_EMAIL=your-email@example.com | |
| STUDENT_API_PORT=8000 | |
| ``` | |
| ### GitHub | |
| ```bash | |
| GITHUB_TOKEN=ghp_your_personal_access_token | |
| GITHUB_USERNAME=your-username | |
| ``` | |
| ### LLM Provider | |
| ```bash | |
| # Choose one | |
| LLM_PROVIDER=anthropic # or openai | |
| ANTHROPIC_API_KEY=sk-ant-... | |
| # OR | |
| OPENAI_API_KEY=sk-... | |
| LLM_MODEL=claude-3-5-sonnet-20241022 | |
| ``` | |
| ### Instructor | |
| ```bash | |
| DATABASE_URL=postgresql://user:password@localhost:5432/llm_deployment | |
| EVALUATION_API_URL=http://your-server:8001/api/evaluate | |
| ``` | |
| ## Usage | |
| ### For Students | |
| 1. **Start the Student API** | |
| ```bash | |
| python -m student.api | |
| ``` | |
| The API will listen on `http://localhost:8000/api/build` | |
| 2. **Test with a sample request** | |
| ```bash | |
| curl -X POST http://localhost:8000/api/build \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "email": "your-email@example.com", | |
| "secret": "your-secret", | |
| "task": "test-task-abc", | |
| "round": 1, | |
| "nonce": "unique-nonce-123", | |
| "brief": "Create a simple Hello World page", | |
| "checks": ["Page displays Hello World"], | |
| "evaluation_url": "http://localhost:8001/api/evaluate", | |
| "attachments": [] | |
| }' | |
| ``` | |
| ### For Instructors | |
| 1. **Start the Evaluation API** | |
| ```bash | |
| python -m instructor.api | |
| ``` | |
| 2. **Prepare submissions.csv** | |
| ```csv | |
| timestamp,email,endpoint,secret | |
| 2025-01-15T10:00:00,student1@example.com,http://student1.com/api/build,secret1 | |
| 2025-01-15T10:05:00,student2@example.com,http://student2.com/api/build,secret2 | |
| ``` | |
| 3. **Run Round 1 task generation** | |
| ```bash | |
| python -m instructor.round1 | |
| ``` | |
| This will: | |
| - Load submissions from CSV | |
| - Generate unique tasks from templates | |
| - POST tasks to student endpoints | |
| - Log results to database | |
| 4. **Run evaluations** | |
| ```bash | |
| python -m instructor.evaluate | |
| ``` | |
| This will: | |
| - Fetch pending submissions | |
| - Clone repositories | |
| - Run static, LLM, and Playwright checks | |
| - Save results to database | |
| 5. **Run Round 2 task generation** | |
| ```bash | |
| python -m instructor.round2 | |
| ``` | |
| This will: | |
| - Find all Round 1 submissions | |
| - Generate Round 2 update tasks | |
| - POST to student endpoints | |
| ## Task Templates | |
| Task templates are YAML files in the `templates/` directory. Example: | |
| ```yaml | |
| id: sum-of-sales | |
| brief: Publish a single-page site that fetches data.csv from attachments... | |
| attachments: | |
| - name: data.csv | |
| url: data:text/csv;base64,placeholder | |
| checks: | |
| - "Page title equals 'Sales Summary {{ seed }}'" | |
| - "Bootstrap 5 CSS loaded from jsdelivr" | |
| round2: | |
| - brief: Add a Bootstrap table #product-sales... | |
| checks: | |
| - "Table #product-sales exists" | |
| ``` | |
| ### Template Variables | |
| - `{{ seed }}`: Unique seed based on email and timestamp | |
| - `{{ hash }}`: Deterministic hash value | |
| - `{{ result }}`: Generated numeric value | |
| ## API Endpoints | |
| ### Student API | |
| **POST /api/build** | |
| - Receives task request | |
| - Returns 200 on acceptance | |
| - Processes in background | |
| **GET /api/status/{task_id}** | |
| - Returns task status | |
| **GET /health** | |
| - Health check | |
| ### Instructor API | |
| **POST /api/evaluate** | |
| - Receives repo submission | |
| - Validates against task record | |
| - Returns 200 on success | |
| **GET /api/submissions/{email}** | |
| - Returns all submissions for email | |
| **GET /api/results/{email}** | |
| - Returns all evaluation results | |
| ## Evaluation Criteria | |
| ### Static Checks | |
| - ✓ Repository created after task sent | |
| - ✓ MIT LICENSE exists in root | |
| - ✓ README.md present with good structure | |
| - ✓ No secrets in git history | |
| ### LLM Checks | |
| - ✓ README.md professional quality (0-1 score) | |
| - ✓ Code quality and best practices (0-1 score) | |
| ### Dynamic Checks | |
| - ✓ Task-specific requirements (from template) | |
| - ✓ Page loads successfully | |
| - ✓ JavaScript evaluations | |
| - ✓ Element presence and content | |
| ## Database Schema | |
| ### Tasks Table | |
| - Task requests sent to students | |
| - Fields: email, task, round, nonce, brief, checks, etc. | |
| ### Repos Table | |
| - Repository submissions from students | |
| - Fields: email, task, round, repo_url, commit_sha, pages_url | |
| ### Results Table | |
| - Evaluation results | |
| - Fields: email, task, round, check, score, reason, logs | |
| ## Troubleshooting | |
| ### Common Issues | |
| **Student API not receiving requests** | |
| - Check firewall settings | |
| - Ensure port 8000 is accessible | |
| - Verify endpoint URL in submissions.csv | |
| **GitHub Pages not deploying** | |
| - Verify GITHUB_TOKEN has repo permissions | |
| - Check repository is public | |
| - Wait up to 60 seconds for Pages to activate | |
| **LLM generation fails** | |
| - Check API key is valid | |
| - Verify API quota/credits | |
| - Review logs for error details | |
| **Playwright tests fail** | |
| - Ensure chromium is installed: `playwright install chromium` | |
| - Check Pages URL is accessible | |
| - Increase timeout if needed | |
| **Database connection errors** | |
| - Verify PostgreSQL is running | |
| - Check DATABASE_URL credentials | |
| - Ensure database exists | |
| ## Development | |
| ### Running Tests | |
| ```bash | |
| pytest tests/ | |
| ``` | |
| ### Code Formatting | |
| ```bash | |
| black . | |
| ruff check . | |
| ``` | |
| ### Type Checking | |
| ```bash | |
| mypy . | |
| ``` | |
| ## License | |
| MIT License | |
| Copyright (c) 2025 | |
| Permission is hereby granted, free of charge, to any person obtaining a copy | |
| of this software and associated documentation files (the "Software"), to deal | |
| in the Software without restriction, including without limitation the rights | |
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | |
| copies of the Software, and to permit persons to whom the Software is | |
| furnished to do so, subject to the following conditions: | |
| The above copyright notice and this permission notice shall be included in all | |
| copies or substantial portions of the Software. | |
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | |
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | |
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | |
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | |
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | |
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | |
| SOFTWARE. | |
| ## Contributing | |
| 1. Fork the repository | |
| 2. Create a feature branch | |
| 3. Make your changes | |
| 4. Submit a pull request | |
| ## Support | |
| For issues and questions: | |
| - Check the troubleshooting section | |
| - Review logs in `logs/app.log` | |
| - Open an issue on GitHub | |