Upload folder using huggingface_hub

AGENTS.md

@@ -1,85 +1,49 @@
-# Autonomy Labs — Agent Notes
+# Repository Guidelines
 
-This repo powers a FastAPI web app deployed to Hugging Face Spaces (Docker). It includes:
-- a Supabase-backed login + user data
-- a chat UI (OpenAI-compatible providers)
-- a PTY-backed web terminal over WebSockets
-- optional Codex CLI/SDK integration
-- optional MCP tooling integration
+## Project Structure & Module Organization
 
-## Non-negotiables
-- **Never commit secrets** (API keys, tokens, cookies, private SSH keys). Use env vars / HF Spaces Secrets.
-- **Assume hostile clients**: UI checks are not security. Dangerous endpoints must be protected server-side.
-- **Prefer minimal, reversible changes** with clear validation steps.
+- `main.py`: app entrypoint (loads dotenv, creates FastAPI app).
+- `app/`: backend package; feature routers live in `app/routes/` (chat, codex, mcp, terminal, user, admin, rag).
+- `static/`: frontend assets (`landing.html`, `index.html`, `dashboard.html` + `dashboard.js`/`dashboard.css`).
+- `tests/`: pytest suite (security-gate coverage is especially important).
+- `docs/`: architecture and ops notes (start with `docs/ARCHITECTURE.md`).
 
-## Repo map
-- `main.py`: FastAPI backend (currently a single large file).
-- `static/index.html`: login page (Supabase Auth).
-- `static/dashboard.html`: main UI (chat, terminal, agent mode, notes, settings UI).
-- `docker-entrypoint.sh`: runtime setup (persistence under `/data`, writes Codex auth files from env if provided).
-- `Dockerfile`: image build (Python + Node + CLIs).
-- `codex_agent.mjs`: Node wrapper around `@openai/codex-sdk`.
-- `.github/workflows/deploy.yml`: deploy to Hugging Face Space.
-- `.github/workflows/codex-autofix.yml`: optional GitHub Action to run Codex for autofixes.
+## Build, Test, and Development Commands
 
-## How to run (local)
-- Python deps: `pip install -r requirements.txt`
-- Start: `uvicorn main:app --host 0.0.0.0 --port 7860`
-- Open: `http://localhost:7860`
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt -r requirements-dev.txt
+uvicorn main:app --reload --host 0.0.0.0 --port 7860
+```
 
-## Key environment variables
-### Supabase
-- `SUPABASE_URL`
-- `SUPABASE_KEY` (anon key used by the frontend)
+- Lint/format: `ruff check .` and `ruff format .`
+- Tests: `pytest -q`
+- Docker (Spaces-like): `docker build -t autonomy-labs . && docker run --rm -p 7860:7860 --env-file .env autonomy-labs`
 
-### Chat defaults (UI convenience)
-- `DEFAULT_BASE_URL` (e.g. `https://router.huggingface.co/v1`)
-- `DEFAULT_API_KEY` (avoid using in production; don’t commit)
-- `DEFAULT_MODEL`
+## Coding Style & Naming Conventions
 
-### Codex (HF Spaces Secrets recommended)
-Supported token env names (either set works):
-- `CODEX_ID_TOKEN` or `ID_TOKEN`
-- `CODEX_ACCESS_TOKEN` or `ACCESS_TOKEN`
-- `CODEX_REFRESH_TOKEN` or `REFRESH_TOKEN`
-- optional: `CODEX_ACCOUNT_ID` or `ACCOUNT_ID`
+- Python 3.11+, 4-space indentation; prefer explicit types for API payloads and settings.
+- Keep modules small: add new routes under `app/routes/<feature>.py` instead of growing large files.
+- Ruff is the source of truth (line length is 120); fix lint before pushing.
 
-At runtime, the container writes:
-- `~/.codex/.auth.json`
-- `~/.codex/auth.json`
+## Testing Guidelines
 
-### Gemini / Claude
-Prefer env-based auth (keep tokens out of the UI and git):
-- Gemini: typically `GEMINI_API_KEY` (or Google GenAI envs, depending on mode)
-- Claude: typically `ANTHROPIC_API_KEY`
+- Use `pytest` and keep tests fast and isolated (set env via `monkeypatch`).
+- Name tests `tests/test_<topic>.py` and add coverage for:
+  - auth enforcement on dangerous endpoints (`/ws/terminal`, `/api/codex*`, `/api/mcp*`)
+  - feature flags (e.g., `ENABLE_CODEX=0` should hard-disable routes).
 
-If adding “Codex-like” auth files for these CLIs, document **exact paths + formats** and keep them **generated at runtime** from Secrets.
+## Commit & Pull Request Guidelines
 
-### SSH (optional)
-Default behavior: container may generate `~/.ssh/id_ed25519` and persist to `/data/.ssh` (Spaces).
-If adding SSH-from-secrets support, prefer:
-- `SSH_PRIVATE_KEY` (+ optional `SSH_PUBLIC_KEY`, `SSH_KNOWN_HOSTS`)
-and ensure files are `0600`, never logged.
+- Commits use short, imperative subjects (e.g., “Add RAG document indexing MVP”), no required prefixes.
+- PRs should include: summary, how you tested (`pytest -q`, `ruff check .`), screenshots for UI changes, and any new env vars documented in `.env.example`.
 
-## Security posture (important)
-The web terminal and any agent/Codex/MCP execution is effectively remote code execution if exposed.
+## Security & Configuration Tips
 
-Before shipping features, ensure:
-- server-side auth checks for `/ws/terminal` and all `/api/codex*` + `/api/mcp*` endpoints
-- explicit capability flags (e.g. `ENABLE_TERMINAL`, `ENABLE_CODEX`, `ENABLE_MCP`) with safe defaults
-- rate limiting / abuse controls if public
-
-## Development hygiene
-- Prefer extracting modules from `main.py` rather than growing it further.
-- Prefer moving large inline JS/CSS out of `static/dashboard.html` into `static/*.js` + `static/*.css`.
-- Keep UI theme tokens consistent between login and dashboard.
-
-## Quick validation
-- Python syntax: `python3 -m py_compile main.py`
-- Basic endpoint check: `curl -sSf http://localhost:7860/health`
+- Never commit secrets; use environment variables / Hugging Face Spaces Secrets.
+- High-risk features are gated by Supabase auth and flags (`ENABLE_TERMINAL`, `ENABLE_CODEX`, `ENABLE_MCP`, `ENABLE_INDEXING`). Keep defaults conservative and document changes in `SECURITY.md`.
 
 ## Deployment notes (HF Spaces)
 - Port: `7860`
 - Persistence: `/data` is used for `~/.codex`, `~/.ssh`, and a default workspace directory when available.
 - Web terminals often require device auth flows; avoid localhost callback assumptions.
-

TASKS.md

@@ -43,7 +43,9 @@ Legend:
 ## P2/P3 — MCP registry
 - [~] First-class MCP registry storage (per-user persistence via backend).
 - [~] Admin-managed MCP templates (server-side persisted).
-- [~] “Test connection”, “list tools”, tool allow/deny UI (SSRF-safe).
+- [~] “Test connection” (browser/CORS; SSRF-safe server proxy pending).
+- [x] “List tools” (via `/api/mcp/tools`).
+- [x] Tool allow/deny policy (UI + server-side enforcement).
 - [x] Import/export `mcp.json` via UI with validation.
 
 ## P3 — RAG + indexing (docs/web/GitHub) + “password manager”

app/mcp_policy.py

@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from fastapi import HTTPException
+
+from app.storage import user_data_dir
+
+
+def _policy_path(user_id: str) -> Path:
+    return user_data_dir(user_id) / "mcp-policy.json"
+
+
+def load_mcp_policy(user_id: str) -> dict[str, Any]:
+    path = _policy_path(user_id)
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+        if not isinstance(data, dict):
+            raise ValueError("Invalid policy")
+        allow = data.get("allow")
+        deny = data.get("deny")
+        return {
+            "version": int(data.get("version") or 1),
+            "allow": allow if isinstance(allow, list) else [],
+            "deny": deny if isinstance(deny, list) else [],
+        }
+    except FileNotFoundError:
+        return {"version": 1, "allow": [], "deny": []}
+    except Exception as e:
+        raise HTTPException(status_code=500, detail={"code": "internal_error", "message": str(e)}) from e
+
+
+def save_mcp_policy(user_id: str, policy: dict[str, Any]) -> dict[str, Any]:
+    allow_in = policy.get("allow")
+    deny_in = policy.get("deny")
+
+    allow = [str(x).strip() for x in (allow_in or []) if str(x).strip()] if isinstance(allow_in, list) else []
+    deny = [str(x).strip() for x in (deny_in or []) if str(x).strip()] if isinstance(deny_in, list) else []
+
+    allow = list(dict.fromkeys(allow))[:500]
+    deny = list(dict.fromkeys(deny))[:500]
+
+    payload = {"version": int(policy.get("version") or 1), "allow": allow, "deny": deny}
+    path = _policy_path(user_id)
+    try:
+        path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
+        return payload
+    except Exception as e:
+        raise HTTPException(status_code=500, detail={"code": "internal_error", "message": str(e)}) from e
+
+
+def tool_allowed(tool_name: str, policy: dict[str, Any]) -> bool:
+    name = (tool_name or "").strip()
+    if not name:
+        return False
+    deny = policy.get("deny") if isinstance(policy.get("deny"), list) else []
+    allow = policy.get("allow") if isinstance(policy.get("allow"), list) else []
+    deny_set = {str(x).strip() for x in deny if str(x).strip()}
+    allow_set = {str(x).strip() for x in allow if str(x).strip()}
+    if name in deny_set:
+        return False
+    if allow_set and name not in allow_set:
+        return False
+    return True
+

app/routes/mcp.py

@@ -4,6 +4,7 @@ from fastapi import APIRouter, HTTPException, Request
 from pydantic import BaseModel
 
 from app.auth import require_user_from_request
+from app.mcp_policy import load_mcp_policy, tool_allowed
 from app.settings import feature_enabled
 
 router = APIRouter()
@@ -13,10 +14,26 @@ router = APIRouter()
 async def mcp_tools_list(http_request: Request):
     if not feature_enabled("mcp"):
         raise HTTPException(status_code=403, detail={"code": "feature_disabled", "message": "MCP is disabled"})
-    _ = await require_user_from_request(http_request)
+    user = await require_user_from_request(http_request)
+    user_id = str(user.get("id") or "")
     try:
+        policy = load_mcp_policy(user_id)
         result = await http_request.app.state.codex_mcp_client.list_tools()
-        return result
+        tools = None
+        if isinstance(result, dict) and isinstance(result.get("tools"), list):
+            tools = result.get("tools")
+        if tools is None:
+            return result
+        filtered = []
+        for t in tools:
+            if not isinstance(t, dict):
+                continue
+            name = str(t.get("name") or "").strip()
+            if tool_allowed(name, policy):
+                filtered.append(t)
+        return {**result, "tools": filtered}
+    except HTTPException:
+        raise
     except Exception as e:
         raise HTTPException(status_code=500, detail={"code": "mcp_error", "message": str(e)}) from e
 
@@ -30,8 +47,17 @@ class McpCallRequest(BaseModel):
 async def mcp_tools_call(request: McpCallRequest, http_request: Request):
     if not feature_enabled("mcp"):
         raise HTTPException(status_code=403, detail={"code": "feature_disabled", "message": "MCP is disabled"})
-    _ = await require_user_from_request(http_request)
+    user = await require_user_from_request(http_request)
+    user_id = str(user.get("id") or "")
     try:
+        policy = load_mcp_policy(user_id)
+        if not tool_allowed(request.name, policy):
+            raise HTTPException(
+                status_code=403,
+                detail={"code": "tool_denied", "message": f"MCP tool not allowed: {request.name}"},
+            )
         return await http_request.app.state.codex_mcp_client.call_tool(request.name, request.arguments)
+    except HTTPException:
+        raise
     except Exception as e:
         raise HTTPException(status_code=500, detail={"code": "mcp_error", "message": str(e)}) from e

app/routes/user.py

@@ -8,6 +8,7 @@ from fastapi import APIRouter, HTTPException, Request
 from pydantic import BaseModel
 
 from app.auth import require_user_from_request
+from app.mcp_policy import load_mcp_policy, save_mcp_policy
 from app.settings import feature_enabled
 from app.storage import user_data_dir
 
@@ -93,3 +94,24 @@ async def put_mcp_registry(body: McpRegistry, http_request: Request):
         return {"ok": True, "count": len(servers)}
     except Exception as e:
         raise HTTPException(status_code=500, detail={"code": "internal_error", "message": str(e)}) from e
+
+
+class McpPolicy(BaseModel):
+    version: int = 1
+    allow: list[str] = []
+    deny: list[str] = []
+
+
+@router.get("/api/user/mcp-policy")
+async def get_mcp_policy(http_request: Request):
+    user = await require_user_from_request(http_request)
+    user_id = str(user.get("id") or "")
+    return load_mcp_policy(user_id)
+
+
+@router.put("/api/user/mcp-policy")
+async def put_mcp_policy(body: McpPolicy, http_request: Request):
+    user = await require_user_from_request(http_request)
+    user_id = str(user.get("id") or "")
+    saved = save_mcp_policy(user_id, body.model_dump())
+    return {"ok": True, "policy": saved}

docs/TROUBLESHOOTING.md

@@ -37,6 +37,7 @@ The Settings → MCP “Test” button runs from your browser, so it is subject
 
 Also note:
 - `mcp.json` import only accepts `http://` / `https://` URLs.
+- If MCP tool calls are blocked, check Settings → MCP → Tool Policy (server-enforced allow/deny list).
 
 ## API errors are shown as JSON
 

static/dashboard.html

@@ -384,6 +384,35 @@
                                 <button onclick="saveMcpRegistryToServer()"
                                     class="bg-gray-700 hover:bg-gray-600 text-white px-2 py-1 rounded text-xs">Save (server)</button>
                             </div>
+                            <div class="bg-gray-900/30 border border-gray-700 rounded-lg p-3 mb-3 space-y-2">
+                                <div class="flex items-center justify-between gap-2">
+                                    <div class="text-xs font-semibold text-gray-300 uppercase">Tool Policy</div>
+                                    <div class="flex gap-2">
+                                        <button onclick="loadMcpPolicyFromServer()"
+                                            class="bg-gray-700 hover:bg-gray-600 text-white px-2 py-1 rounded text-xs">Load</button>
+                                        <button onclick="saveMcpPolicyToServer()"
+                                            class="bg-blue-600 hover:bg-blue-700 text-white px-2 py-1 rounded text-xs">Save</button>
+                                    </div>
+                                </div>
+                                <div class="text-xs text-gray-400">Optional allow/deny list enforced server-side for <code>/api/mcp/call</code> and tool listing.</div>
+                                <div class="grid grid-cols-1 md:grid-cols-2 gap-3">
+                                    <div>
+                                        <label class="block text-xs font-semibold text-gray-400 mb-1 uppercase">Allow tools (one per line)</label>
+                                        <textarea id="mcp-allow-tools" rows="3"
+                                            class="w-full bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500 font-mono"
+                                            placeholder="filesystem.read\nfilesystem.write"></textarea>
+                                        <div class="text-xs text-gray-500 mt-1">If non-empty, only these tools are allowed.</div>
+                                    </div>
+                                    <div>
+                                        <label class="block text-xs font-semibold text-gray-400 mb-1 uppercase">Deny tools (one per line)</label>
+                                        <textarea id="mcp-deny-tools" rows="3"
+                                            class="w-full bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500 font-mono"
+                                            placeholder="filesystem.write"></textarea>
+                                        <div class="text-xs text-gray-500 mt-1">Always blocked (overrides allow).</div>
+                                    </div>
+                                </div>
+                                <div id="mcp-policy-status" class="text-xs text-gray-500"></div>
+                            </div>
                             <div class="flex gap-2 mb-3">
                                 <input id="mcp-direct-url" type="text" placeholder="Direct MCP URL (https://...)"
                                     class="flex-grow bg-gray-700 text-sm rounded border border-gray-600 p-2 text-white outline-none focus:border-blue-500">

static/dashboard.js

@@ -659,6 +659,8 @@ let supabase;
                 initProviderPresets();
                 // Best-effort: hydrate server-persisted MCP registry if local storage is empty.
                 maybeHydrateMcpRegistryFromServer();
+                writeMcpPolicyToInputs(loadMcpPolicyFromLocalStorage());
+                maybeHydrateMcpPolicyFromServer();
                 // Restore last chat mode (chat/autonomous)
                 const savedMode = localStorage.getItem('chat_mode_v1') || 'chat';
                 setChatMode(savedMode);
@@ -2144,6 +2146,7 @@ let supabase;
 
         // --- MCP Admin ---
         const MCP_STORAGE_KEY = 'mcp_servers_v1';
+        const MCP_POLICY_STORAGE_KEY = 'mcp_policy_v1';
 
         async function maybeHydrateMcpRegistryFromServer() {
             try {
@@ -2153,6 +2156,81 @@ let supabase;
             } catch { }
         }
 
+        async function maybeHydrateMcpPolicyFromServer() {
+            try {
+                const raw = localStorage.getItem(MCP_POLICY_STORAGE_KEY);
+                if (raw && raw.trim() && raw.trim() !== '{}' && raw.trim() !== 'null') return;
+                await loadMcpPolicyFromServer({ quiet: true });
+            } catch { }
+        }
+
+        function setMcpPolicyStatus(text) {
+            const el = document.getElementById('mcp-policy-status');
+            if (el) el.textContent = text || '';
+        }
+
+        function readMcpPolicyFromInputs() {
+            const allowEl = document.getElementById('mcp-allow-tools');
+            const denyEl = document.getElementById('mcp-deny-tools');
+            const splitLines = (v) => (v || '').split('\n').map((s) => s.trim()).filter(Boolean);
+            const allow = splitLines(allowEl?.value || '');
+            const deny = splitLines(denyEl?.value || '');
+            return { version: 1, allow, deny };
+        }
+
+        function writeMcpPolicyToInputs(policy) {
+            const allowEl = document.getElementById('mcp-allow-tools');
+            const denyEl = document.getElementById('mcp-deny-tools');
+            const allow = Array.isArray(policy?.allow) ? policy.allow : [];
+            const deny = Array.isArray(policy?.deny) ? policy.deny : [];
+            if (allowEl) allowEl.value = allow.join('\n');
+            if (denyEl) denyEl.value = deny.join('\n');
+        }
+
+        function loadMcpPolicyFromLocalStorage() {
+            try {
+                return JSON.parse(localStorage.getItem(MCP_POLICY_STORAGE_KEY) || '{}');
+            } catch {
+                return {};
+            }
+        }
+
+        function saveMcpPolicyToLocalStorage(policy) {
+            localStorage.setItem(MCP_POLICY_STORAGE_KEY, JSON.stringify(policy || {}));
+        }
+
+        async function loadMcpPolicyFromServer({ quiet = false } = {}) {
+            try {
+                if (!quiet) setMcpPolicyStatus('Loading policy from server...');
+                const res = await authFetch('/api/user/mcp-policy');
+                if (!res.ok) throw new Error(await res.text());
+                const data = await res.json();
+                saveMcpPolicyToLocalStorage(data);
+                writeMcpPolicyToInputs(data);
+                if (!quiet) setMcpPolicyStatus('Loaded policy from server.');
+            } catch (e) {
+                if (!quiet) setMcpPolicyStatus(`Failed to load policy: ${e?.message || e}`);
+            }
+        }
+
+        async function saveMcpPolicyToServer() {
+            try {
+                setMcpPolicyStatus('Saving policy to server...');
+                const policy = readMcpPolicyFromInputs();
+                const res = await authFetch('/api/user/mcp-policy', {
+                    method: 'PUT',
+                    headers: { 'Content-Type': 'application/json' },
+                    body: JSON.stringify(policy),
+                });
+                if (!res.ok) throw new Error(await res.text());
+                const data = await res.json();
+                saveMcpPolicyToLocalStorage(data?.policy || policy);
+                setMcpPolicyStatus('Saved policy to server.');
+            } catch (e) {
+                setMcpPolicyStatus(`Failed to save policy: ${e?.message || e}`);
+            }
+        }
+
         async function loadMcpRegistryFromServer({ quiet = false } = {}) {
             const statusEl = document.getElementById('codex-mcp-list');
             try {