feat: Add multi-provider support for agent generation with model validation, new templates, and security checks.

.env.example

@@ -1,8 +1,5 @@
-ANTHROPIC_API_KEY=your_api_key_here
+# Anthropic API Key
+ANTHROPIC_API_KEY=your_anthropic_api_key_here
 
-# Available Models:
-# - claude-3-5-sonnet-20241022 (Latest Sonnet - Recommended)
-# - claude-3-opus-20240229 (Opus - Most Powerful)
-# - claude-3-sonnet-20240229 (Previous Sonnet)
-# - claude-3-haiku-20240307 (Haiku - Fastest/Cheapest)
-ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
+# Hugging Face API Token
+HUGGINGFACEHUB_API_TOKEN=your_huggingface_token_here

GEMINI.md

@@ -1,3 +1,4 @@
+
 # LLMAgentBuilder Context
 
 ## Project Overview

frontend/src/components/AgentForm.jsx

@@ -5,15 +5,29 @@ const AgentForm = ({ onGenerate, isLoading }) => {
         name: '',
         prompt: '',
         task: '',
-        model: 'claude-3-5-sonnet-20241022'
+        provider: 'anthropic',
+        model: 'claude-3-5-sonnet-20241022',
+        stream: false
     });
 
     const handleChange = (e) => {
-        const { name, value } = e.target;
-        setFormData(prev => ({
-            ...prev,
-            [name]: value
-        }));
+        const { name, value, type, checked } = e.target;
+        setFormData(prev => {
+            const newData = {
+                ...prev,
+                [name]: type === 'checkbox' ? checked : value
+            };
+
+            // Reset model when provider changes
+            if (name === 'provider' && value !== prev.provider) {
+                if (value === 'anthropic') {
+                    newData.model = 'claude-3-5-sonnet-20241022';
+                } else if (value === 'huggingface') {
+                    newData.model = 'meta-llama/Meta-Llama-3-8B-Instruct';
+                }
+            }
+            return newData;
+        });
     };
 
     const handleSubmit = (e) => {
@@ -38,6 +52,19 @@ const AgentForm = ({ onGenerate, isLoading }) => {
                     />
                 </div>
 
+                <div className="form-group">
+                    <label htmlFor="provider">Provider</label>
+                    <select
+                        id="provider"
+                        name="provider"
+                        value={formData.provider}
+                        onChange={handleChange}
+                    >
+                        <option value="anthropic">Anthropic</option>
+                        <option value="huggingface">Hugging Face</option>
+                    </select>
+                </div>
+
                 <div className="form-group">
                     <label htmlFor="model">Model</label>
                     <select
@@ -46,9 +73,19 @@ const AgentForm = ({ onGenerate, isLoading }) => {
                         value={formData.model}
                         onChange={handleChange}
                     >
-                        <option value="claude-3-5-sonnet-20241022">Claude 3.5 Sonnet (Latest)</option>
-                        <option value="claude-3-opus-20240229">Claude 3 Opus</option>
-                        <option value="claude-3-haiku-20240307">Claude 3 Haiku</option>
+                        {formData.provider === 'anthropic' ? (
+                            <>
+                                <option value="claude-3-5-sonnet-20241022">Claude 3.5 Sonnet (Latest)</option>
+                                <option value="claude-3-5-haiku-20241022">Claude 3.5 Haiku</option>
+                                <option value="claude-3-opus-20240229">Claude 3 Opus</option>
+                                <option value="claude-3-haiku-20240307">Claude 3 Haiku (Legacy)</option>
+                            </>
+                        ) : (
+                            <>
+                                <option value="meta-llama/Meta-Llama-3-8B-Instruct">Meta Llama 3 8B Instruct</option>
+                                <option value="mistralai/Mistral-7B-Instruct-v0.3">Mistral 7B Instruct v0.3</option>
+                            </>
+                        )}
                     </select>
                 </div>
 
@@ -78,6 +115,18 @@ const AgentForm = ({ onGenerate, isLoading }) => {
                     />
                 </div>
 
+                <div className="form-group checkbox-group">
+                    <label>
+                        <input
+                            type="checkbox"
+                            name="stream"
+                            checked={formData.stream}
+                            onChange={handleChange}
+                        />
+                        Stream Response
+                    </label>
+                </div>
+
                 <button type="submit" className="btn-primary" disabled={isLoading}>
                     {isLoading ? 'Generating...' : 'Generate Agent'}
                 </button>

implementation_plan.md

@@ -1,50 +1,107 @@
-# Implementation Plan - Hugging Face Spaces Deployment
+# Configure Model Options
 
 ## Goal Description
 
-Prepare the LLMAgentBuilder for deployment to **Hugging Face Spaces** (Docker). This involves making the application stateless (serverless-friendly) and packaging both the React frontend and FastAPI backend into a single Docker container.
+Update the agent generation form to include the latest Anthropic models and add support for Hugging Face models. Additionally, implement backend validation, streaming support, security measures, a sandbox executor, and observability.
 
 ## User Review Required
 >
-> [!IMPORTANT]
-> **File Saving Change**: The application will no longer save files to the *server's* `generated_agents/` directory. Instead, it will trigger a **file download** in the user's browser. This is necessary because Hugging Face Spaces have ephemeral storage (files are lost on restart).
+> [!NOTE]
+> I am adding `claude-3-5-haiku-20241022` to the list.
+> I am also adding support for Hugging Face models (via `huggingface_hub`).
+> I am adding backend validation using Pydantic.
+> I am adding a streaming response toggle.
+> I am adding security measures (pre-commit hook).
+> I am adding a Sandbox Executor for safe agent execution.
+> I am adding Prometheus metrics and a `/healthz` endpoint.
 
 ## Proposed Changes
 
-### 1. Make Application Stateless
+### Frontend
 
-#### [MODIFY] [server/main.py](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/server/main.py)
+#### [MODIFY] [AgentForm.jsx](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/frontend/src/components/AgentForm.jsx)
 
-- Remove `os.makedirs` and file writing logic.
-- Keep the return of the `code` string.
+- Add a "Provider" dropdown (Anthropic, Hugging Face).
+- Update "Model" dropdown options based on the selected provider.
+- Add `claude-3-5-haiku-20241022` for Anthropic.
+- Add `meta-llama/Meta-Llama-3-8B-Instruct` and `mistralai/Mistral-7B-Instruct-v0.3` for Hugging Face.
+- Add a "Stream Response" checkbox (default: false).
+- Add a "Test Agent" button to execute the generated code in the sandbox.
+- Display execution results (output/errors) in the UI.
 
-#### [MODIFY] [frontend/src/App.jsx](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/frontend/src/App.jsx)
+### Backend
 
-- Update `handleGenerate` to receive the code and trigger a browser download of the `.py` file.
-- Remove "Saved to path" message.
+#### [NEW] [models.py](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/server/models.py)
 
-### 2. Single-Container Setup (FastAPI + React)
+- Define `ProviderEnum` (Anthropic, HuggingFace).
+- Define `GenerateRequest` Pydantic model with validation:
+  - `provider`: ProviderEnum
+  - `model`: Validated against an allowlist per provider.
+  - `stream`: bool (default: False)
 
-#### [MODIFY] [server/main.py](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/server/main.py)
+#### [MODIFY] [main.py](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/server/main.py)
 
-- Mount `StaticFiles` to serve the React `dist/` folder.
-- Add a catch-all route to serve `index.html` for React routing.
+- Import `GenerateRequest` from `models.py`.
+- Update `generate_agent` endpoint to use the new validation model.
+- Add `/api/execute` endpoint.
+- Add `prometheus-fastapi-instrumentator`.
+- Add `/metrics` and `/healthz` endpoints.
 
-### 3. Docker Configuration
+#### [MODIFY] [agent_builder.py](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/llm_agent_builder/agent_builder.py)
 
-#### [NEW] [Dockerfile](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/Dockerfile)
+- Update `build_agent` to accept `provider` and `stream` arguments.
+- Select the appropriate template (`agent_template.py.j2` or `agent_template_hf.py.j2`) based on the provider.
+- Pass `stream` to the template context.
 
-- **Stage 1 (Frontend)**: Node.js image -> `npm run build`.
-- **Stage 2 (Backend)**: Python image -> Install requirements -> Copy React build -> Run Uvicorn.
+#### [NEW] [agent_template_hf.py.j2](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/llm_agent_builder/templates/agent_template_hf.py.j2)
+
+- Create a new Jinja2 template for agents using `huggingface_hub.InferenceClient`.
+- Use `HUGGINGFACEHUB_API_TOKEN` for authentication.
+- Implement conditional logic for `stream=True` vs `stream=False`.
+
+#### [MODIFY] [agent_template.py.j2](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/llm_agent_builder/templates/agent_template.py.j2)
+
+- Fix the environment variable name from `GEMINI_API_KEY` to `ANTHROPIC_API_KEY`.
+
+### Sandbox Executor
+
+#### [NEW] [sandbox.py](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/server/sandbox.py)
+
+- Implement `run_in_sandbox(code: str, task: str) -> str`:
+  - Write code to a temporary file.
+  - Use `subprocess.Popen` to execute the file.
+  - Use `preexec_fn` to set `resource.setrlimit`:
+    - `RLIMIT_CPU`: Limit CPU time (e.g., 30 seconds).
+    - `RLIMIT_AS`: Limit address space (memory) (e.g., 512MB).
+  - Capture `stdout` and `stderr`.
+  - Handle timeouts and errors.
+
+### Security & Misc
+
+#### [NEW] [.env.example](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/.env.example)
+
+- Add `ANTHROPIC_API_KEY` and `HUGGINGFACEHUB_API_TOKEN` placeholders.
+
+#### [NEW] [pre-commit-check.sh](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/pre-commit-check.sh)
+
+- Simple script to grep for potential API keys in staged files.
+
+#### [MODIFY] [requirements.txt](file:///wsl.localhost/Ubuntu/root/LLMAgentBuilder/requirements.txt)
+
+- Add `prometheus-fastapi-instrumentator`.
 
 ## Verification Plan
 
+### Automated Tests
+
+- Run `pytest` to ensure no regressions in `agent_builder`.
+
 ### Manual Verification
 
-1. **Local Docker Test**:
-    - Build image: `docker build -t llm-agent-builder .`
-    - Run: `docker run -p 7860:7860 llm-agent-builder`
-    - Access `http://localhost:7860`.
-    - Generate an agent -> Confirm file downloads to local computer.
-2. **Hugging Face Deployment** (User Action):
-    - User pushes `Dockerfile` to their Space.
+- Start the frontend (`npm run dev` in `frontend/`).
+- Verify the new provider and model options.
+- Verify the stream toggle works.
+- Generate an agent with streaming enabled and check the code.
+- Test the "Test Agent" button with a simple task.
+- Verify `/metrics` and `/healthz` endpoints.
+- Try to commit a file with a fake API key to test the hook.

llm_agent_builder/agent_builder.py

@@ -9,19 +9,25 @@ class AgentBuilder:
         self.env = Environment(loader=FileSystemLoader(template_path))
         self.template = self.env.get_template('agent_template.py.j2')
 
-    def build_agent(self, agent_name: str, prompt: str, example_task: str, model: str = "claude-3-5-sonnet-20241022") -> str:
+    def build_agent(self, agent_name: str, prompt: str, example_task: str, model: str = "claude-3-5-sonnet-20241022", provider: str = "anthropic", stream: bool = False) -> str:
         """
         Generates the Python code for a new agent.
 
         :param agent_name: The name of the agent class to be generated.
         :param prompt: The system prompt for the agent.
         :param example_task: An example task for the agent.
-        :param model: The default Anthropic model to use.
+        :param model: The model to use.
+        :param provider: The provider (anthropic or huggingface).
+        :param stream: Whether to stream the response.
         :return: The generated Python code as a string.
         """
-        return self.template.render(
+        template_name = 'agent_template_hf.py.j2' if provider == 'huggingface' else 'agent_template.py.j2'
+        template = self.env.get_template(template_name)
+        
+        return template.render(
             agent_name=agent_name,
             prompt=prompt,
             example_task=example_task,
-            model=model
+            model=model,
+            stream=stream
         )

llm_agent_builder/templates/agent_template.py.j2

@@ -31,9 +31,9 @@ if __name__ == '__main__':
     args = parser.parse_args()
 
     # Ensure API key is set
-    api_key = os.environ.get("GEMINI_API_KEY")
+    api_key = os.environ.get("ANTHROPIC_API_KEY")
     if not api_key:
-        raise ValueError("GEMINI_API_KEY environment variable not set. Please set it in your .env file or environment.")
+        raise ValueError("ANTHROPIC_API_KEY environment variable not set. Please set it in your .env file or environment.")
 
     try:
         agent = {{ agent_name }}(api_key=api_key)

llm_agent_builder/templates/agent_template_hf.py.j2

@@ -0,0 +1,71 @@
+import os
+from huggingface_hub import InferenceClient
+
+class {{ agent_name }}:
+    def __init__(self, api_key):
+        self.client = InferenceClient(token=api_key)
+        self.prompt = "{{- prompt -}}"
+
+    def run(self, task):
+        messages = [
+            {"role": "system", "content": self.prompt},
+            {"role": "user", "content": task}
+        ]
+        
+        {% if stream %}
+        response = self.client.chat_completion(
+            model="{{ model }}",
+            messages=messages,
+            max_tokens=1024,
+            stream=True
+        )
+        full_response = ""
+        for chunk in response:
+            content = chunk.choices[0].delta.content
+            if content:
+                print(content, end="", flush=True)
+                full_response += content
+        print() # Newline after stream
+        return full_response
+        {% else %}
+        response = self.client.chat_completion(
+            model="{{ model }}",
+            messages=messages,
+            max_tokens=1024,
+            stream=False
+        )
+        return response.choices[0].message.content
+        {% endif %}
+
+if __name__ == '__main__':
+    import os
+    import argparse
+    from dotenv import load_dotenv
+
+    load_dotenv()
+
+    # Parse command line arguments
+    parser = argparse.ArgumentParser(description="Run the {{ agent_name }} agent.")
+    parser.add_argument("--task", default="{{- example_task -}}", help="The task to be performed by the agent")
+    args = parser.parse_args()
+
+    # Ensure API key is set
+    api_key = os.environ.get("HUGGINGFACEHUB_API_TOKEN")
+    if not api_key:
+        raise ValueError("HUGGINGFACEHUB_API_TOKEN environment variable not set. Please set it in your .env file or environment.")
+
+    try:
+        agent = {{ agent_name }}(api_key=api_key)
+        print(f"Running {{ agent_name }} with task: {args.task}\n")
+        result = agent.run(args.task)
+        print("Response:")
+        print("-" * 50)
+        # If streaming, result is already printed, but we print it again or just the separator?
+        # The template logic above prints during stream. 
+        # If not streaming, we print result.
+        {% if not stream %}
+        print(result)
+        {% endif %}
+        print("-" * 50)
+    except Exception as e:
+        print(f"Error running agent: {e}")

pre-commit-check.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# Pre-commit hook to check for exposed API keys
+
+# Keywords to search for
+KEYWORDS="ANTHROPIC_API_KEY HUGGINGFACEHUB_API_TOKEN sk-ant- hf_"
+
+# Check staged files
+FILES=$(git diff --cached --name-only)
+
+if [ -z "$FILES" ]; then
+    exit 0
+fi
+
+FOUND_KEYS=0
+
+for FILE in $FILES; do
+    # Skip check for .env.example and pre-commit-check.sh itself
+    if [[ "$FILE" == ".env.example" || "$FILE" == "pre-commit-check.sh" ]]; then
+        continue
+    fi
+
+    for KEYWORD in $KEYWORDS; do
+        if grep -q "$KEYWORD" "$FILE"; then
+            echo "ERROR: Found potential API key or sensitive keyword '$KEYWORD' in '$FILE'."
+            FOUND_KEYS=1
+        fi
+    done
+done
+
+if [ $FOUND_KEYS -eq 1 ]; then
+    echo "Commit rejected. Please remove sensitive information before committing."
+    exit 1
+fi
+
+exit 0

server/main.py

@@ -11,6 +11,8 @@ sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from llm_agent_builder.agent_builder import AgentBuilder
 
+from server.models import GenerateRequest, ProviderEnum
+
 app = FastAPI()
 
 # Configure CORS
@@ -22,12 +24,6 @@ app.add_middleware(
     allow_headers=["*"],
 )
 
-class GenerateRequest(BaseModel):
-    name: str
-    prompt: str
-    task: str
-    model: str = "claude-3-5-sonnet-20241022"
-
 @app.post("/api/generate")
 async def generate_agent(request: GenerateRequest):
     try:
@@ -36,7 +32,9 @@ async def generate_agent(request: GenerateRequest):
             agent_name=request.name,
             prompt=request.prompt,
             example_task=request.task,
-            model=request.model
+            model=request.model,
+            provider=request.provider,
+            stream=request.stream
         )
         
         # Stateless: Return code directly, do not save to disk

server/models.py

@@ -0,0 +1,36 @@
+from enum import Enum
+from pydantic import BaseModel, validator
+from typing import Optional
+
+class ProviderEnum(str, Enum):
+    ANTHROPIC = "anthropic"
+    HUGGINGFACE = "huggingface"
+
+class GenerateRequest(BaseModel):
+    name: str
+    prompt: str
+    task: str
+    provider: ProviderEnum = ProviderEnum.ANTHROPIC
+    model: str
+    stream: bool = False
+
+    @validator('model')
+    def validate_model(cls, v, values):
+        provider = values.get('provider')
+        if provider == ProviderEnum.ANTHROPIC:
+            allowed = [
+                "claude-3-5-sonnet-20241022",
+                "claude-3-opus-20240229",
+                "claude-3-haiku-20240307",
+                "claude-3-5-haiku-20241022"
+            ]
+            if v not in allowed:
+                raise ValueError(f"Model {v} not supported for Anthropic")
+        elif provider == ProviderEnum.HUGGINGFACE:
+            allowed = [
+                "meta-llama/Meta-Llama-3-8B-Instruct",
+                "mistralai/Mistral-7B-Instruct-v0.3"
+            ]
+            if v not in allowed:
+                raise ValueError(f"Model {v} not supported for Hugging Face")
+        return v

task.md

@@ -0,0 +1,22 @@
+# Tasks
+
+- [x] Professionalize CLI (Previous Context)
+- [x] Implement React + Vite Frontend <!-- id: 0 -->
+- [x] Implement FastAPI Backend <!-- id: 1 -->
+- [x] Deploy to Hugging Face Spaces <!-- id: 2 -->
+- [/] Configure Model Options <!-- id: 3 -->
+  - [ ] Update AgentForm.jsx with new models and provider selector
+  - [ ] Add Stream Response toggle
+  - [/] Implement Backend Validation (Pydantic)
+  - [/] Implement Streaming Support in Templates
+  - [ ] Add Security Measures (.env.example, pre-commit)
+  - [ ] Add Security Measures (.env.example, pre-commit)
+  - [ ] Update AgentBuilder default if needed
+- [ ] Implement Sandbox Executor <!-- id: 4 -->
+  - [ ] Create Sandbox module (subprocess + resource limits)
+  - [ ] Add /api/execute endpoint
+  - [ ] Add 'Test Agent' button in Frontend
+- [ ] Add Observability <!-- id: 5 -->
+  - [ ] Add Prometheus Metrics
+  - [ ] Add /healthz endpoint
+- [x] Push to GitHub