docs/frameworks/EMBEDDED-SERVICES.md

---
title: "Embedded Services"
description: "Reference for 9Router and CLIProxyAPI"
---

# Embedded Services

> **Version:** v3.8.4
> **Last updated:** 2026-05-25
> **Audience:** Engineers adding, maintaining, or debugging embedded services (9Router, CLIProxyAPI).

Embedded services are locally-installed process sidecar tools that OmniRoute installs, supervises, and
exposes as first-class routing targets. Unlike external providers (which are reached over the internet
via API keys), embedded services run on the same machine as OmniRoute and communicate over loopback.

---

## Table of Contents

1. [Overview](#1-overview)
2. [Architecture — 4 layers](#2-architecture--4-layers)
3. [Lifecycle state machine](#3-lifecycle-state-machine)
4. [API reference](#4-api-reference)
5. [Security](#5-security)
6. [Adding a new embedded service](#6-adding-a-new-embedded-service)
7. [Troubleshooting](#7-troubleshooting)
8. [FAQ](#8-faq)

---

## 1. Overview

### Why embedded services?

Two services are embedded as of v3.8.4:

| Service         | npm package                                    | Default port | Purpose                                                                                              |
| --------------- | ---------------------------------------------- | :----------: | ---------------------------------------------------------------------------------------------------- |
| **9Router**     | `9router`                                      |    20130     | AI router that OmniRoute can use as a sub-provider. Models exposed as `9router/{sub}/{model}`        |
| **CLIProxyAPI** | `@anthropic/cli-proxy` (via `cliproxy` binary) |     auto     | Local proxy adapter for Anthropic CLI auth flows. Provides fallback routing when OAuth tokens expire |

Both follow the same supervisory model:

- OmniRoute installs them under `DATA_DIR/services/{name}/` (isolated from OmniRoute's own `package.json`)
- OmniRoute spawns and monitors them as child processes
- OmniRoute injects an ephemeral API key into the child's environment and rotates it without downtime
- All management routes (`/api/services/*`) are **LOCAL_ONLY** — accessible only from loopback (hard rule #17)

### Key decisions (from design plan)

| Decision                              | Value                                                                    |
| ------------------------------------- | ------------------------------------------------------------------------ |
| Dashboard access to 9Router native UI | Reverse proxy at `/dashboard/providers/services/9router/embed/*`         |
| Installation mechanism                | `npm install {package}` via `execFile` (no shell interpolation)          |
| Consumption mode                      | Provider registered as `9router/{sub}/{model}` in routing engine         |
| API key management                    | OmniRoute generates, encrypts at-rest (AES-256-GCM), and injects via env |
| Dashboard location                    | `/dashboard/providers/services` (two tabs)                               |
| Auto-start                            | Toggle per service, default OFF                                          |

---

## 2. Architecture — 4 layers

```
┌────────────────────────────────────────────────────────────────────┐
│  Layer 1 — UI                                                      │
│  /dashboard/providers/services  (tabs: CLIProxyAPI | 9Router)      │
│  Logs live (SSE), Start/Stop/Restart/Update, Settings, Install     │
│                                                                    │
│  src/app/(dashboard)/dashboard/providers/services/                 │
│    ├── page.tsx               Shell + tab routing by ?tab=         │
│    ├── tabs/                  CliproxyServiceTab, NinerouterServiceTab│
│    └── components/            ServiceStatusCard, ServiceLifecycleButtons,│
│                               ServiceLogsPanel, ApiKeyCard, ...    │
└──────────────────────┬─────────────────────────────────────────────┘
                       │ HTTP (Next.js fetch)
┌──────────────────────▼─────────────────────────────────────────────┐
│  Layer 2 — API (LOCAL_ONLY — loopback only)                        │
│                                                                    │
│  /api/services/9router/{install|start|stop|restart|update|         │
│                          rotate-key|status|auto-start|logs}        │
│  /api/services/cliproxy/{install|start|stop|restart|update|        │
│                           status|auto-start|logs}                  │
│  /dashboard/providers/services/9router/embed/[...path]             │
│    (reverse HTTP + WebSocket proxy → 9Router upstream)             │
│                                                                    │
│  Gate: LOCAL_ONLY_API_PREFIXES includes "/api/services/" and       │
│        "/dashboard/providers/services/*/embed/"                    │
└──────────────────────┬─────────────────────────────────────────────┘
                       │ in-process calls
┌──────────────────────▼─────────────────────────────────────────────┐
│  Layer 3 — ServiceSupervisor (src/lib/services/)                   │
│                                                                    │
│  ServiceSupervisor.ts   Generic supervisor (child_process.spawn)   │
│    ├── install:    execFile('npm', ['install', pkg, '--prefix'])    │
│    ├── start:      spawn(node, [entrypoint], {env, cwd})           │
│    ├── api_key:    crypto.randomBytes(32) → env NINEROUTER_API_KEY  │
│    ├── port:       20130 for 9Router (configurable)                │
│    ├── logs:       stdio ring buffer 5 MB → SSE events             │
│    ├── health:     HTTP GET /health every 2–5 s, lazy recovery     │
│    └── lifecycle:  SIGTERM 15 s → SIGKILL                          │
│                                                                    │
│  registry.ts        getSupervisor(name) / registerSupervisor()     │
│  bootstrap.ts       Bootstraps all SERVICES[] at process start     │
│  apiKey.ts          getOrCreateApiKey(), generateServiceApiKey()   │
│  modelSync.ts       Periodic GET /v1/models → service_models table │
│  ringBuffer.ts      Circular log buffer (5 MB per service)         │
│  healthCheck.ts     Polling HTTP health probe                      │
│  installers/        ninerouter.ts, cliproxy.ts (installer adapters)│
└──────────────────────┬─────────────────────────────────────────────┘
                       │ OpenAI-compatible HTTP (loopback)
┌──────────────────────▼─────────────────────────────────────────────┐
│  Layer 4 — Provider / Routing                                      │
│                                                                    │
│  open-sse/executors/ninerouter.ts                                  │
│    Re-looks up port and API key per-request (no caching).          │
│    Strips "9router/" prefix from model id before proxying.         │
│    Returns 503 service_not_running if supervisor not in "running". │
│                                                                    │
│  src/shared/constants/providers.ts                                 │
│    Entry for "9router": isEmbeddedService: true                    │
│                                                                    │
│  open-sse/config/providerRegistry.ts                               │
│    Models stored as "9router/{sub}/{model}" (prefixed).            │
│    Synced every 5 min by modelSync.ts.                             │
└────────────────────────────────────────────────────────────────────┘
```

### Key source files

| File                                        | Role                                             |
| ------------------------------------------- | ------------------------------------------------ |
| `src/lib/services/ServiceSupervisor.ts`     | Core class: lifecycle, lock, health, ring buffer |
| `src/lib/services/bootstrap.ts`             | Process-level registration and auto-start        |
| `src/lib/services/registry.ts`              | Singleton map `tool → supervisor`                |
| `src/lib/services/apiKey.ts`                | Key generation, AES-256-GCM encryption at-rest   |
| `src/lib/services/modelSync.ts`             | Periodic model sync (5 min) + on-demand          |
| `src/lib/services/ringBuffer.ts`            | 5 MB circular log buffer with SSE subscribe      |
| `src/lib/services/healthCheck.ts`           | HTTP health probe (configurable interval)        |
| `src/lib/services/installers/ninerouter.ts` | npm install/update/uninstall for 9Router         |
| `src/lib/services/installers/cliproxy.ts`   | npm install/update/uninstall for CLIProxyAPI     |
| `src/app/api/services/9router/_lib.ts`      | `getOrInitSupervisor()` helper                   |
| `src/app/api/services/[name]/logs/route.ts` | Shared SSE logs endpoint                         |
| `open-sse/executors/ninerouter.ts`          | Provider executor (Layer 4)                      |

---

## 3. Lifecycle state machine

```
                    install()
  ┌─────────────┐ ──────────► ┌─────────────┐
  │ not_installed│             │   stopped   │◄──────────────────┐
  └─────────────┘             └──────┬──────┘                   │
                                     │ start()                   │
                                     ▼                           │ stop()
                               ┌──────────┐                      │
                               │ starting │                      │
                               └────┬─────┘                     │
                  health probe ok   │         crash / SIGTERM    │
                               ┌────▼─────┐  (exit within 5s)   │
                               │ running  │──── crash ──────────►┤
                               └────┬─────┘                   ┌─▼────┐
                             stop() │                          │error │
                                    ▼                          └──────┘
                               ┌──────────┐
                               │ stopping │
                               └──────────┘
```

States stored in the `version_manager` DB table (`status` column) and mirrored
in `ServiceSupervisor` in-memory state. The in-memory state is authoritative for
a running process; the DB state is the durable fallback at boot.

### State transitions

| From            | Event                              | To                     |
| --------------- | ---------------------------------- | ---------------------- |
| `not_installed` | `install()` succeeds               | `stopped`              |
| `stopped`       | `start()` called                   | `starting`             |
| `starting`      | health probe returns 200           | `running`              |
| `starting`      | process exits before healthy       | `error`                |
| `running`       | `stop()` called                    | `stopping` → `stopped` |
| `running`       | process exits unexpectedly (< 5 s) | `error` (fast crash)   |
| `running`       | process exits unexpectedly (> 5 s) | `error`                |
| `error`         | `start()` called                   | `starting`             |
| any             | `stop()` while `stopping`          | no-op                  |

### Operation lock

`ServiceSupervisor` serializes lifecycle operations through an async operation lock
(`withLock()`). Concurrent `start()` calls on the same supervisor result in exactly
one spawn; the second caller waits and returns the existing status. This prevents
race conditions when, for example, auto-start and a UI button fire simultaneously.

---

## 4. API reference

All routes under `/api/services/` are **LOCAL_ONLY** (loopback only, hard rule #17).
Non-loopback requests receive `403 LOCAL_ONLY` regardless of auth token.

### 4.1 9Router endpoints (8 routes)

#### `POST /api/services/9router/install`

Install 9Router from npm. Creates `DATA_DIR/services/9router/` with its own
`package.json` and `node_modules/`. Does not conflict with OmniRoute's own deps.

**Request body** (all optional):

```json
{ "version": "latest" }
```

| Field     | Type     | Default    | Description                          |
| --------- | -------- | ---------- | ------------------------------------ |
| `version` | `string` | `"latest"` | npm version tag or semver to install |

**Responses:**

| Status | Description                                            |
| ------ | ------------------------------------------------------ |
| `200`  | `{ ok: true, installedVersion: "x.y.z", path: "..." }` |
| `400`  | Invalid request body (Zod validation failure)          |
| `409`  | Already installing (lock held)                         |
| `500`  | npm install failed — see `message` for friendly error  |

**Notes:** Uses `execFile('npm', [...])` — no shell, no interpolation (hard rule #13).
EACCES errors are surfaced as friendly messages.

---

#### `POST /api/services/9router/start`

Start 9Router. Registers a supervisor if not already registered, then calls
`supervisor.start()`. Idempotent when already running.

**Request body:** none

**Responses:**

| Status | Description                                          |
| ------ | ---------------------------------------------------- |
| `200`  | `ServiceStatus` object (see schema below)            |
| `409`  | 9Router is not installed (`status: "not_installed"`) |
| `503`  | Start failed (process error — see `lastError`)       |

**ServiceStatus schema:**

```json
{
  "tool": "9router",
  "state": "running",
  "pid": 12345,
  "port": 20130,
  "health": "healthy",
  "startedAt": "2026-05-25T10:00:00.000Z",
  "lastError": null
}
```

---

#### `POST /api/services/9router/stop`

Gracefully stop 9Router. Sends SIGTERM, waits 15 s, then SIGKILL if still alive.
Idempotent when already stopped.

**Request body:** none

**Responses:**

| Status | Description                        |
| ------ | ---------------------------------- |
| `200`  | `ServiceStatus` (state: "stopped") |
| `503`  | Stop failed unexpectedly           |

---

#### `POST /api/services/9router/restart`

Equivalent to `stop()` then `start()` under the operation lock.

**Request body:** none

**Responses:** same as `start` (returns final `ServiceStatus`).

---

#### `POST /api/services/9router/update`

Updates 9Router to a newer npm version. If the service is running, it is stopped
first, npm install is run (installing the newer version in-place), and then the
service is restarted.

**Request body** (all optional):

```json
{ "version": "latest" }
```

**Responses:**

| Status | Description                                                     |
| ------ | --------------------------------------------------------------- |
| `200`  | `{ ok: true, previousVersion: "...", installedVersion: "..." }` |
| `400`  | Invalid body                                                    |
| `500`  | npm update failed                                               |

---

#### `POST /api/services/9router/rotate-key`

Generates a new API key for 9Router, encrypts it at-rest, and restarts the service
(if running) so it picks up the new key from its environment. The old key is
invalidated immediately.

**Request body:** none

**Responses:**

| Status | Description                                |
| ------ | ------------------------------------------ |
| `200`  | `{ keyRotated: true, restarted: boolean }` |
| `500`  | Rotation failed                            |

**Security:** The new key is never returned in the response (no credential leak).
It is stored encrypted (AES-256-GCM) in the `version_manager` table.

---

#### `GET /api/services/9router/status`

Returns combined live + DB status including version metadata and API key preview.

**Responses:**

| Status | Description        |
| ------ | ------------------ |
| `200`  | See schema below   |
| `500`  | Status read failed |

**Response schema:**

```json
{
  "tool": "9router",
  "state": "running",
  "pid": 12345,
  "port": 20130,
  "health": "healthy",
  "startedAt": "2026-05-25T10:00:00.000Z",
  "lastError": null,
  "installedVersion": "1.2.3",
  "latestVersion": "1.2.4",
  "updateAvailable": true,
  "apiKeyMasked": "nr_****abcd",
  "autoStart": false,
  "providerExpose": false
}
```

---

#### `POST /api/services/9router/auto-start`

Toggle the auto-start flag. When `enabled: true`, the service starts automatically
the next time OmniRoute boots (if the service is installed).

**Request body:**

```json
{ "enabled": true }
```

**Responses:**

| Status | Description           |
| ------ | --------------------- |
| `200`  | `{ autoStart: true }` |
| `400`  | Invalid body          |

---

#### `GET /api/services/9router/logs`

SSE stream of live logs from 9Router's stdout/stderr ring buffer.

**Query parameters:**

| Param    | Type      | Default | Description                                               |
| -------- | --------- | ------- | --------------------------------------------------------- |
| `tail`   | `integer` | 200     | How many historical lines to send first (max 1000)        |
| `filter` | `string`  | none    | Case-insensitive substring filter (no regex — ReDoS-safe) |

**SSE events:**

| Event       | Data        | Description             |
| ----------- | ----------- | ----------------------- |
| `snapshot`  | `LogLine[]` | Initial historical tail |
| `log`       | `LogLine`   | Live log line           |
| `heartbeat` | `{}`        | Keep-alive every 15 s   |

**LogLine schema:**

```json
{ "ts": 1716633600000, "stream": "stdout", "line": "[9router] Listening on :20130" }
```

**Responses:**

| Status | Description                                   |
| ------ | --------------------------------------------- |
| `200`  | `text/event-stream`                           |
| `400`  | `filter` parameter too long (> 200 chars)     |
| `404`  | Service not found (supervisor not registered) |

---

### 4.2 CLIProxyAPI endpoints (7 routes)

CLIProxyAPI has the same endpoint shape as 9Router minus `rotate-key` (CLIProxyAPI
does not require an injected API key; it authenticates via the host's existing CLI
config) and `status` includes fewer fields.

| Method | Path                                | Description                          |
| ------ | ----------------------------------- | ------------------------------------ |
| `POST` | `/api/services/cliproxy/install`    | Install CLIProxyAPI from npm         |
| `POST` | `/api/services/cliproxy/start`      | Start CLIProxyAPI                    |
| `POST` | `/api/services/cliproxy/stop`       | Stop CLIProxyAPI                     |
| `POST` | `/api/services/cliproxy/restart`    | Restart CLIProxyAPI                  |
| `POST` | `/api/services/cliproxy/update`     | Update to newer version              |
| `GET`  | `/api/services/cliproxy/status`     | Live + DB status (no `apiKeyMasked`) |
| `POST` | `/api/services/cliproxy/auto-start` | Toggle auto-start                    |

The shared `GET /api/services/{name}/logs` endpoint (see §4.1) works for both
services using the `[name]` dynamic segment.

---

### 4.3 Reverse proxy (9Router dashboard embed)

The dashboard embeds the 9Router web UI inside an iframe via an internal reverse
proxy at:

```
GET|POST|... /dashboard/providers/services/9router/embed/[...path]
```

This proxy:

- Forwards the request to `http://127.0.0.1:{port}/{path}` (loopback only)
- Strips incoming `cookie` and `authorization` headers (no leakage of OmniRoute session)
- Injects `Authorization: Bearer {apiKey}` for 9Router authentication
- Strips `set-cookie`, `content-security-policy`, `x-frame-options`, `cross-origin-*` from the response
- Rewrites HTML responses to inject `<base href>` and normalize absolute paths (`/foo` → `/dashboard/.../embed/foo`)

WebSocket upgrades for the embedded dashboard are handled by a companion server on a
dedicated port (see `src/lib/services/embedWsProxy.ts`).

**Security:** The embed proxy routes are classified under `LOCAL_ONLY_API_PREFIXES`
and can only be reached from loopback. An attacker who obtains a JWT via a
Cloudflare/Ngrok tunnel cannot proxy into embedded services.

---

## 5. Security

### LOCAL_ONLY enforcement (hard rule #17)

All routes under `/api/services/` and `/dashboard/providers/services/*/embed/` are
classified as LOCAL_ONLY in `src/server/authz/routeGuard.ts`. The loopback check
runs unconditionally before any auth branch:

```
request arrives
  → isLocalOnlyPath(path)?
      → non-loopback → 403 LOCAL_ONLY (always, before auth check)
      → loopback    → fall through to normal auth
```

This prevents a leaked JWT (e.g., via a tunnel) from triggering `npm install` or
process spawning. See `docs/security/ROUTE_GUARD_TIERS.md` for the full tier
matrix.

### API key injection

9Router requires an API key for its own HTTP endpoints. OmniRoute:

1. Generates a key via `crypto.randomBytes(32).toString("base64url")` with a
   service-specific prefix (`nr_` for 9Router).
2. Encrypts it at-rest using AES-256-GCM (same cipher used for provider credentials).
3. Decrypts and injects it as `NINEROUTER_API_KEY` environment variable at spawn time.
4. Never returns the plaintext key in any HTTP response.

### SSRF defense

The reverse HTTP proxy (`/dashboard/.../embed/[...path]`) is hardcoded to forward
only to `http://127.0.0.1:{port}`. It never follows redirects to non-loopback
destinations. The `ssrf-req-filter` library is used to reject any upstream URL that
resolves outside the loopback range.

### Shell safety (hard rule #13)

`npm install` is invoked via `execFile('npm', ['install', pkg, '--prefix', dir])` —
no template literals, no shell, no interpolation of external paths into the command
string. Runtime values (ports, API keys) are passed via the child's `env` object.

### Error sanitization (hard rule #12)

All error responses from `/api/services/*` go through `buildErrorBody()` or
`sanitizeErrorMessage()`. Raw `err.stack` and `err.message` are never returned
verbatim to the caller.

---

## 6. Adding a new embedded service

Follow these 8 steps. Read the existing implementations in `src/lib/services/installers/`
and `src/app/api/services/` as the canonical reference.

### Step 1 — Create the installer

Create `src/lib/services/installers/{name}.ts` modeled on `ninerouter.ts`:

```typescript
export const NAME_PACKAGE = "your-npm-package";
export const NAME_DEFAULT_PORT = 20132; // pick a free port

export async function install(version = "latest"): Promise<InstallResult> { ... }
export async function update(version = "latest"): Promise<InstallResult> { ... }
export async function uninstall(): Promise<void> { ... }
export function resolveSpawnArgs(apiKey: string, port: number): SpawnArgs { ... }
export async function getInstalledVersion(): Promise<string | null> { ... }
export async function getLatestVersion(): Promise<string | null> { ... }
```

Use `runNpm(['install', NAME_PACKAGE, '--prefix', dir])` from `installers/utils.ts`
— never `execSync` or shell interpolation.

### Step 2 — Register in bootstrap

Add a `ServiceEntry` to the `SERVICES` array in `src/lib/services/bootstrap.ts`:

```typescript
{
  tool: "myservice",
  port: NAME_DEFAULT_PORT,
  healthPath: "/health",
  healthIntervalMs: 5_000,
  stopTimeoutMs: 15_000,
  logsBufferBytes: 5_242_880,
  needsApiKey: true, // false if no API key needed
}
```

Extend `buildSpawnArgsFactory()` to handle `cfg.tool === "myservice"`.

### Step 3 — Add migration and DB seed

Ensure the service has a row in `version_manager` via a migration in
`src/lib/db/migrations/`. The row should have:

```sql
INSERT OR IGNORE INTO version_manager (tool, status, auto_start, provider_expose)
VALUES ('myservice', 'not_installed', 0, 0);
```

### Step 4 — Create the 7 API endpoints

Under `src/app/api/services/{name}/`:

```
_lib.ts            getOrInitSupervisor() helper
install/route.ts   POST — calls installer.install()
start/route.ts     POST — calls supervisor.start()
stop/route.ts      POST — calls supervisor.stop()
restart/route.ts   POST — calls supervisor.restart()
update/route.ts    POST — calls installer.update()
status/route.ts    GET  — merges live + DB status
auto-start/route.ts POST — toggles auto_start flag
```

The shared `GET /api/services/[name]/logs` route is already wired — no changes
needed there.

Delegate all error responses through `createErrorResponse()` / `buildErrorBody()`.

### Step 5 — Add to LOCAL_ONLY_API_PREFIXES

In `src/server/authz/routeGuard.ts`, verify that `/api/services/` is already listed.
If you introduce a new prefix (e.g., `/api/tools/`), add it to both
`LOCAL_ONLY_API_PREFIXES` and, if it spawns processes, to `SPAWN_CAPABLE_PREFIXES`.
Add a test in `tests/unit/authz/routeGuard.test.ts`.

### Step 6 — Add the UI tab

Create `src/app/(dashboard)/dashboard/providers/services/tabs/{Name}ServiceTab.tsx`.
Reuse shared components:

- `ServiceStatusCard` — live state + health badge
- `ServiceLifecycleButtons` — Start / Stop / Restart / Update
- `ServiceLogsPanel` — SSE log tail (connects to `/api/services/{name}/logs`)
- `ApiKeyCard` — key reveal + rotate (if `needsApiKey: true`)

Register the tab in `ServicesPageShell.tsx`.

### Step 7 — Add the provider entry (if the service is a routing target)

If the embedded service exposes an OpenAI-compatible `/v1/chat/completions` endpoint:

1. Add a provider entry in `src/shared/constants/providers.ts` with `isEmbeddedService: true`.
2. Create `open-sse/executors/{name}.ts` extending `BaseExecutor`. Re-lookup port and
   API key per-request (never cache in the constructor). Return a `503 service_not_running`
   response when the supervisor state is not `"running"`.
3. Register models in `open-sse/config/providerRegistry.ts` with the service prefix
   (e.g., `myservice/sub/model`). `modelSync.ts` will keep them updated.

### Step 8 — Document and test

1. Update `docs/frameworks/EMBEDDED-SERVICES.md` (this file) — add the service to the
   table in §1 and any new endpoints to §4.
2. Add unit tests in `tests/unit/services/` (lifecycle, installer, API shape).
3. Add integration test in `tests/integration/services/` (behind `RUN_SERVICES_INT=1`).
4. Update `docs/openapi.yaml` with the new endpoints.

---

## 7. Troubleshooting

### Service does not start

**Symptoms:** Start button returns 503, state stays `"error"` or `"starting"`.

**Checklist:**

1. Check `GET /api/services/{name}/logs` (or the Logs panel in the dashboard). Look
   for lines like `Error: ENOENT`, `address already in use`, or `Cannot find module`.
2. Verify `npm` is in PATH: `which npm` from the same user account that runs OmniRoute.
3. Verify the service is installed: check `GET /api/services/{name}/status` for
   `installedVersion`. If `null`, run install first.
4. Check `DATA_DIR/services/{name}/node_modules/` exists and is not empty.
5. Check the `lastError` field in the status response for the sanitized exit reason.

---

### Cold start is slow (> 10 s to reach `running`)

**Symptoms:** State stays `"starting"` for a long time before going to `"running"` or `"error"`.

**Explanation:** 9Router's cold start includes importing large dependency trees (DNS,
tunnel, MITM modules). Default health interval is 2 s with 3 attempts before the
supervisor declares a timeout (but continues polling).

**Fix:** The `healthIntervalMs` and the `waitForHealthy` timeout
(`healthIntervalMs * 3`) are configurable in `bootstrap.ts`. For services with longer
startup times, increase `healthIntervalMs` to 5000 and `stopTimeoutMs` to 30 000.

---

### Port collision (`EADDRINUSE`)

**Symptoms:** Logs show `address already in use :::20130`.

**Causes:**

- Another process is already using port 20130.
- A previous 9Router process was not fully stopped (zombie PID).

**Fix:**

1. Change the default port via `NINEROUTER_PORT` environment variable in `.env`.
2. Find and kill the conflicting process: `lsof -ti :20130 | xargs kill -9`.
3. The port is configurable per service in `bootstrap.ts` via the `port` field.

**Note:** 9Router defaults to port 20130 specifically to avoid colliding with
OmniRoute's default port 20128.

---

### Permission denied (EACCES) on install

**Symptoms:** Install returns 500, logs show `EACCES` or `permission denied`.

**Causes:**

- `DATA_DIR` or its parent is not writable by the OmniRoute process.
- Running inside Docker rootless without write access to the mapped volume.

**Fix:**

1. Check `DATA_DIR` (default: `~/.omniroute/`): `ls -la ~/.omniroute/`
2. Ensure the OmniRoute process user owns the directory: `chown -R $USER ~/.omniroute/`
3. In Docker, ensure the volume mount has the correct permissions for the container user.

---

### Update fails (`npm install` timeout or network error)

**Symptoms:** Update returns 500 with `InstallError`, logs show network timeout.

**Checklist:**

1. Confirm npm registry is reachable: `npm ping`.
2. Check for corporate proxy: `npm config get proxy`, `npm config get https-proxy`.
3. Try the install manually: `npm install {package}@latest --prefix ~/.omniroute/services/{name}/`.
4. If behind an air-gap, pre-download the tarball and use `npm install /path/to/tarball.tgz`.

---

### Service shows `"error"` state immediately after start (fast crash)

**Symptoms:** State transitions from `"starting"` to `"error"` in under 5 seconds.
`lastError` shows `"Fast crash (exited with code 1)"`.

**Checklist:**

1. Read the full log tail: `GET /api/services/{name}/logs?tail=500`.
2. Common cause: missing environment variables expected by the service.
3. For 9Router: verify `NINEROUTER_DISABLE_MITM=true` and
   `NINEROUTER_DISABLE_TUNNEL=true` are in the env passed at spawn (see
   `installers/ninerouter.ts` `resolveSpawnArgs`).

---

## 8. FAQ

**Q: Can I expose the embedded services endpoints to non-loopback clients?**

No. The LOCAL_ONLY tier is intentional (hard rule #17). Routes that can invoke
`npm install` or spawn `node` processes must not be reachable from non-loopback
traffic, because a leaked JWT via a tunnel (Cloudflare, Ngrok, Tailscale) would
otherwise allow arbitrary process spawning. There is no opt-out carve-out for
`/api/services/` — unlike `/api/mcp/`, it is excluded from the manage-scope bypass
list. See `docs/security/ROUTE_GUARD_TIERS.md`.

---

**Q: Will 9Router and CLIProxyAPI be available in production/cloud deployments?**

Yes. Both services follow the same local-first model as OmniRoute itself. They run
on the same machine and communicate over loopback. "Production" here means the VPS
or local server where OmniRoute is deployed, not a remote cloud provider.

---

**Q: How do I debug the supervisor?**

1. Tail the SSE log stream: `curl -N http://localhost:20128/api/services/9router/logs`.
2. Check structured logs in OmniRoute's pino output filtered by
   `service:supervisor` namespace.
3. Inspect the DB row: `sqlite3 ~/.omniroute/omniroute.db "SELECT * FROM version_manager WHERE tool='9router'"`.
4. Use `GET /api/services/9router/status` to see the current live state, PID, health,
   and `lastError` in one call.

---

**Q: The supervisor shows `health: "degraded"` or `health: "unknown"` but state is `"running"`. Is that a problem?**

`"degraded"` means the health probe returned a non-200 response. `"unknown"` means no
probe has completed yet (race with first poll). Both are transient during startup.
If health stays `"degraded"` for more than `healthIntervalMs * 3` ms after
`"running"`, the embedded service is running but its HTTP API is not responding. Check
whether the port is correct in the status response and whether the service is actually
listening on that port.

---

**Q: Can I change the 9Router API key without a full restart?**

No. The API key is passed to 9Router via an environment variable at spawn time.
Environment variables cannot be changed in a running process. `POST .../rotate-key`
automatically stops and restarts the service to apply the new key. The key rotation
takes effect within the service's `stopTimeoutMs` (default 15 s) plus its startup
time.

---

**Q: What is the ring buffer limit and what happens when it fills?**

Each service has a dedicated 5 MB ring buffer. When the buffer is full, the oldest
log lines are evicted to make room for new ones. The SSE `snapshot` event returns
the most recent lines within the `tail` limit. Logs are not persisted to disk unless
`logsBufferPath` is set in the DB row.

---

## See also

- `docs/security/ROUTE_GUARD_TIERS.md` — LOCAL_ONLY tier details
- `docs/architecture/CODEBASE_DOCUMENTATION.md` — §3.2 Embedded Services module mapping
- `docs/architecture/ARCHITECTURE.md` — system-level context
- `docs/openapi.yaml` — machine-readable endpoint definitions
- `CLAUDE.md` §"Adding a New Embedded Service" — quick-reference checklist