Spaces:
Running
Running
| title: Agent UI | |
| emoji: 🤖 | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: docker | |
| pinned: false | |
| header: mini | |
| # Agent UI | |
| A multi-agent AI interface with code execution, web search, image generation, and deep research — all orchestrated from a single command center. | |
| ## Local Install | |
| ```bash | |
| pip install . # Install from pyproject.toml | |
| python -m backend.main # Start server at http://localhost:8765 | |
| ``` | |
| Or use Make shortcuts: | |
| ```bash | |
| make install # pip install . | |
| make dev # Start dev server | |
| ``` | |
| Configure API keys in the Settings panel, or set environment variables: | |
| | Variable | Purpose | | |
| |----------|---------| | |
| | `LLM_API_KEY` | Default LLM provider token (any OpenAI-compatible API) | | |
| | `HF_TOKEN` | HuggingFace token (image generation, hosted models) | | |
| | `E2B_API_KEY` | [E2B](https://e2b.dev) sandbox for code execution | | |
| | `SERPER_API_KEY` | [Serper](https://serper.dev) for web search | | |
| ## Docker | |
| ```bash | |
| docker build -t agent-ui . | |
| docker run -p 7860:7860 -e LLM_API_KEY=... agent-ui | |
| ``` | |
| CLI options: `--port`, `--no-browser`, `--config-dir`, `--workspace-dir`, `--multi-user`. | |
| For HuggingFace Spaces deployment, set `HF_BUCKET` and `HF_BUCKET_TOKEN` secrets for workspace persistence across restarts. | |
| ## Architecture | |
| ``` | |
| backend/ | |
| ├── agents.py # Agent registry (single source of truth) + shared LLM utilities | |
| ├── main.py # FastAPI routes, SSE streaming, file management | |
| ├── command.py # Command center: tool routing, agent launching | |
| ├── code.py # Code agent: E2B sandbox execution | |
| ├── agent.py # Web agent: search + browse | |
| ├── research.py # Research agent: multi-source deep analysis | |
| ├── image.py # Image agent: generate/edit via HuggingFace | |
| └── tools.py # Direct tools (execute_code, web_search, show_html, etc.) | |
| frontend/ | |
| ├── index.html # Entry point | |
| ├── utils.js # Global state, shared helpers (setupInputListeners, closeAllPanels) | |
| ├── timeline.js # Sidebar timeline data + rendering | |
| ├── sessions.js # Session CRUD + panel | |
| ├── tabs.js # Tab creation/switching, sendMessage | |
| ├── streaming.js # SSE streaming, code cells, action widgets, markdown | |
| ├── workspace.js # Workspace serialize/restore | |
| ├── settings.js # Settings CRUD, themes, debug/files/sessions panels | |
| ├── app.js # Initialization, event listeners, DOMContentLoaded | |
| ├── style.css # All styles (CSS custom properties for theming) | |
| └── research-ui.js # Research-specific UI components | |
| ``` | |
| ### How It Works | |
| 1. The **command center** receives user messages and decides whether to answer directly or launch sub-agents | |
| 2. Sub-agents (code, web, research, image) run in their own tabs with specialized tools | |
| 3. All communication uses **SSE streaming** — agents yield JSON events with a `type` field | |
| 4. Settings store providers, models, and agent-to-model assignments — any OpenAI-compatible API works | |
| ## Extending Agent UI | |
| ### Adding a New Agent | |
| Only the backend needs changes — the frontend fetches the registry from `GET /api/agents` at startup. | |
| **1. Backend registry** — add to `AGENT_REGISTRY` in `backend/agents.py`: | |
| ```python | |
| "my_agent": { | |
| "label": "MY AGENT", | |
| "system_prompt": "You are a helpful assistant...", | |
| "tool": { | |
| "type": "function", | |
| "function": { | |
| "name": "launch_my_agent", | |
| "description": "Launch my agent for X tasks.", | |
| "parameters": { | |
| "type": "object", | |
| "properties": { | |
| "task": {"type": "string", "description": "The task"}, | |
| "task_id": {"type": "string", "description": "2-3 word ID"} | |
| }, | |
| "required": ["task", "task_id"] | |
| } | |
| } | |
| }, | |
| "tool_arg": "task", | |
| "has_counter": True, | |
| "in_menu": True, | |
| "in_launcher": True, | |
| "placeholder": "Enter message...", | |
| "capabilities": "Short description of what this agent can do.", | |
| }, | |
| ``` | |
| **2. Backend streaming handler** — create `backend/my_agent.py`: | |
| ```python | |
| from .agents import call_llm | |
| def stream_my_agent(client, model, messages, extra_params=None, abort_event=None): | |
| """Generator yielding SSE event dicts.""" | |
| debug_call_number = 0 | |
| while not_done: | |
| # call_llm handles retries and emits debug events | |
| response = None | |
| for event in call_llm(client, model, messages, tools=MY_TOOLS, | |
| extra_params=extra_params, abort_event=abort_event, | |
| call_number=debug_call_number): | |
| if "_response" in event: | |
| response = event["_response"] | |
| debug_call_number = event["_call_number"] | |
| else: | |
| yield event | |
| if event.get("type") in ("error", "aborted"): | |
| return | |
| # Process response, yield events... | |
| yield {"type": "thinking", "content": "..."} | |
| yield {"type": "result", "content": "Final answer"} | |
| yield {"type": "done"} | |
| ``` | |
| Required events: `done`, `error`. Common: `thinking`, `content`, `result`, `result_preview`. | |
| **3. Wire the route** — in `backend/main.py`, add to the streaming handler dispatch (search for `agent_type`): | |
| ```python | |
| elif request.agent_type == "my_agent": | |
| return StreamingResponse(stream_my_agent_handler(...), ...) | |
| ``` | |
| **4. Frontend** — no changes needed. The frontend fetches the registry from `GET /api/agents` at startup. | |
| ### Adding a Direct Tool | |
| Direct tools execute synchronously in the command center (no sub-agent spawned). Only two files need changes. | |
| **1. Define the tool schema + execute function** in `backend/tools.py`: | |
| ```python | |
| my_tool = { | |
| "type": "function", | |
| "function": { | |
| "name": "my_tool", | |
| "description": "Does something useful.", | |
| "parameters": { | |
| "type": "object", | |
| "properties": { | |
| "input": {"type": "string", "description": "The input"} | |
| }, | |
| "required": ["input"] | |
| } | |
| } | |
| } | |
| def execute_my_tool(input: str, files_root: str = None) -> dict: | |
| return {"content": "Result text for the LLM", "extra_data": "..."} | |
| ``` | |
| **2. Register it** in `DIRECT_TOOL_REGISTRY` at the bottom of `backend/tools.py`: | |
| ```python | |
| DIRECT_TOOL_REGISTRY = { | |
| "show_html": { ... }, # existing | |
| "my_tool": { | |
| "schema": my_tool, | |
| "execute": lambda args, ctx: execute_my_tool( | |
| args.get("input", ""), files_root=ctx.get("files_root") | |
| ), | |
| }, | |
| } | |
| ``` | |
| That's it — `command.py` automatically picks up tools from the registry. | |
| ### Modifying System Prompts | |
| All system prompts live in `backend/agents.py` inside `AGENT_REGISTRY`. Edit the `"system_prompt"` field for any agent. | |
| The `get_system_prompt()` function adds dynamic context automatically: | |
| - `{tools_section}` — replaced with available agent descriptions (command center only) | |
| - Current date is appended to all prompts | |
| - Project file tree is appended (in `main.py` wrapper) | |
| - Theme/styling context is added for code agents | |
| ### Adding a Model Provider | |
| In the Settings panel, models are configured through **Providers** and **Models**: | |
| 1. **Add a provider**: name + OpenAI-compatible endpoint URL + API token | |
| 2. **Add a model**: name + provider + API model ID (e.g., `gpt-4o`, `claude-sonnet-4-20250514`) | |
| 3. **Assign models**: pick which model each agent type uses | |
| Any OpenAI-compatible API works (OpenAI, Anthropic via proxy, Ollama, vLLM, etc.). | |
| Settings are stored in `workspace/settings.json` and managed via the Settings panel in the UI. | |
| ### Creating a Theme | |
| Themes are CSS custom property sets defined in `frontend/settings.js`. | |
| **Add to `themeColors` object** (search for `const themeColors`): | |
| ```javascript | |
| myTheme: { | |
| border: '#8e24aa', | |
| bg: '#f3e5f5', | |
| hoverBg: '#e1bee7', | |
| accent: '#6a1b9a', | |
| accentRgb: '106, 27, 154', | |
| ...lightSurface // Use for light themes | |
| }, | |
| ``` | |
| For dark themes, override the surface colors instead of spreading `lightSurface`: | |
| ```javascript | |
| myDarkTheme: { | |
| border: '#bb86fc', | |
| bg: '#1e1e2e', | |
| hoverBg: '#2a2a3e', | |
| accent: '#bb86fc', | |
| accentRgb: '187, 134, 252', | |
| bgPrimary: '#121218', | |
| bgSecondary: '#1e1e2e', | |
| bgTertiary: '#0e0e14', | |
| bgInput: '#0e0e14', | |
| bgHover: '#2a2a3e', | |
| bgCard: '#1e1e2e', | |
| textPrimary: '#e0e0e0', | |
| textSecondary: '#999999', | |
| textMuted: '#666666', | |
| borderPrimary: '#333344', | |
| borderSubtle: '#222233' | |
| }, | |
| ``` | |
| The theme automatically appears in the Settings theme picker — no other changes needed. The `applyTheme()` function reads all properties from the object and sets the corresponding CSS variables. | |
| **Available CSS variables:** `--theme-accent`, `--theme-accent-rgb`, `--theme-bg`, `--theme-hover-bg`, `--theme-border`, `--bg-primary`, `--bg-secondary`, `--bg-tertiary`, `--bg-input`, `--bg-hover`, `--bg-card`, `--text-primary`, `--text-secondary`, `--text-muted`, `--border-primary`, `--border-subtle`. | |
| ## SSE Event Protocol | |
| All agents communicate via Server-Sent Events. Each event is a JSON object with a `type` field. | |
| | Event | Description | | |
| |-------|-------------| | |
| | `done` | Stream complete (required) | | |
| | `error` | `{content}` — error message (required) | | |
| | `thinking` | `{content}` — reasoning text | | |
| | `content` | `{content}` — streamed response tokens | | |
| | `result` | `{content, figures?}` — final output for command center | | |
| | `result_preview` | Same as result, shown inline | | |
| | `retry` | `{attempt, max_attempts, delay, message}` — retrying | | |
| | `debug_call_input` | `{call_number, messages}` — LLM input (debug panel) | | |
| | `debug_call_output` | `{call_number, response}` — LLM output (debug panel) | | |
| | `launch` | `{agent_type, initial_message, task_id}` — spawn sub-agent | | |
| | `tool_start` | `{tool, args}` — direct tool started | | |
| | `tool_result` | `{tool, result}` — direct tool completed | | |
| | `code_start` | `{code}` — code execution started | | |
| | `code` | `{output, error, images}` — code execution result | | |
| ## Key Patterns & Conventions | |
| ### Backend | |
| - **Single source of truth**: `AGENT_REGISTRY` in `backend/agents.py` defines all agent types. The frontend fetches it via `GET /api/agents` — never duplicate agent definitions. | |
| - **LLM calls**: Always use `call_llm()` from `agents.py` — it handles retries, abort checking, and emits `debug_call_input`/`debug_call_output` events for the debug panel. | |
| - **Streaming pattern**: Agent handlers are sync generators yielding event dicts. `_stream_sync_generator()` in `main.py` wraps them for async SSE delivery — never duplicate the async queue boilerplate. | |
| - **Direct tools**: `DIRECT_TOOL_REGISTRY` in `tools.py` maps tool name → `{schema, execute}`. `command.py` dispatches automatically. | |
| - **Result nudging**: When an agent finishes without `<result>` tags, `nudge_for_result()` in `agents.py` asks the LLM for a final answer. It uses `call_llm` internally. | |
| ### Frontend | |
| - **No build system**: Plain `<script>` tags in `index.html`, no bundler. Files share `window` scope. | |
| - **Load order matters**: `utils.js` loads first (declares all globals), then other files. Cross-file function calls are fine because they happen at runtime, not parse time. | |
| - **Global state** lives in `utils.js`: `AGENT_REGISTRY`, `settings`, `activeTabId`, `tabCounter`, `timelineData`, `debugHistory`, `globalFigureRegistry`, etc. | |
| - **Shared helpers** (also in `utils.js`): | |
| - `setupInputListeners(container, tabId)` — wires textarea auto-resize, Enter-to-send, send button click | |
| - `setupCollapseToggle(cell, labelSelector)` — wires click-to-collapse on tool/code cells | |
| - `closeAllPanels()` — closes all right-side panels (settings, debug, files, sessions) | |
| - **Markdown rendering**: `parseMarkdown()` in `streaming.js` is the single entry point (marked + KaTeX + Prism). | |
| - **Panel toggle pattern**: Call `closeAllPanels()` first, then add `.active` to the panel being opened. | |
| - **Workspace persistence**: Changes auto-save via `saveWorkspaceDebounced()`. Tab state is serialized to JSON and posted to `/api/workspace`. | |
| - **Cache busting**: Bump `?v=N` query params in `index.html` when changing JS/CSS files. | |
| ### Naming | |
| - Backend: `stream_<agent>_execution()` for the sync generator, `_stream_<agent>_inner()` for the async wrapper in `main.py` | |
| - Frontend: Agent types use short keys (`code`, `agent`, `research`, `image`, `command`) | |
| - CSS: `--theme-*` for accent colors, `--bg-*` / `--text-*` / `--border-*` for surface colors | |
| ## Verification | |
| Verify backend imports: `python -c "from backend.command import stream_command_center"` | |