docs/DESIGN.md

# Loom Model Panel: A Privacy-First Gateway for Uncorrelated Frontier-Model Judgment on Free Tiers

*System design document — 2026-06. All claims cross-checked against the code at the
commit that ships this file; constants quoted are the env-overridable defaults.*

## 1. Problem & motivation

A single LLM reviewing its own plans inherits its own blind spots. The highest-leverage
use of *other* frontier models is as **uncorrelated second opinions**: a panel of
models with different training lineages (Kimi, DeepSeek, GLM, Nemotron, Llama,
MiniMax…) judging one artifact produces failure-mode diversity a single model cannot.

Three constraints shape the system:

1. **Cost: zero.** Every model must come from recurring-free tiers (NVIDIA NIM free
   credits, Cloudflare Workers AI, GitHub Models, OpenRouter `:free`). Free tiers are
   throttled, flaky, and quota-capped — so reliability must be *engineered around* the
   providers, not assumed from them.
2. **Privacy: per-user, verifiable.** The deployment model is **one instance per user,
   the user's own provider keys, nothing shared** — privacy by isolation rather than by
   access control. On top, a request-level **privacy router** refuses providers that
   train on or log prompts.
3. **Operations: a single ephemeral container.** The target platform (Hugging Face
   Spaces, free Docker tier) gives one container that is recycled on every deploy.
   Anything that must survive — async jobs, metrics, templates, cache — needs explicit
   durability machinery.

The result is a stdlib-only Python service (plus `httpx` and optionally
`huggingface_hub`) exposing a token-guarded HTTP API: one-shot panels
(`/api/critique`, `/api/panel`), whole-repo review (`files`/`repo_url` forms), and
multi-stage orchestrated pipelines (`/api/run` + templates).

## 2. Architecture overview

```
client ──HTTP──▶ BoundedThreadingHTTPServer (MAX_WORKERS=48, REQUEST_TIMEOUT_S=30)
                   │  bearer auth (constant-time), body cap 8MB, 503/429 backpressure
                   ▼
             request builders (panel / critique / run)
              ├─ repopack: {files|repo_url} → packed blob + per-judge context budgeting
              ├─ privacy router: strict | fallback | off
              └─ prompt cache (exact-match, per-space)
                   │
        ┌──────────┴───────────┐
   sync (asyncio.run)     async JobRunner (one background event loop)
        │                      │  durable: JobStore (atomic file + HF Dataset mirror)
        ▼                      ▼  reload-on-boot; MAX_INFLIGHT_JOBS=200
              route_judge / orchestrator stages
                   │  per logical model:
                   ▼
             SlotScheduler ──▶ provider hosts (NVIDIA / Cloudflare / GitHub / OpenRouter)
              rotation · pacing · cooldown/blacklist · cross-provider failover
                   │
                   ▼  every call → per-profile MetricStore (HF-mirrored)
```

Two execution planes share one substrate. The **synchronous plane** runs a request to
completion on a bounded HTTP worker via `asyncio.run`. The **asynchronous plane**
(`{"async": true}`) hands work to a single long-lived background event loop in
`JobRunner`, returns a `job_id` immediately, and the client polls `GET /api/jobs/<id>`
for partial results. Both planes call the same `route_judge`/scheduler/metrics core, so
routing, privacy, caching, and telemetry behave identically regardless of plane.

The canonical request is for a **logical model** (e.g. `kimi-k2.6`); the scheduler maps
it to an ordered list of physical `(provider, model)` hosts and bounces across them on
failure. Catalogs (`providers_catalog.json`, `models_catalog.json`, `benchmarks.json`,
`reasoning_catalog.json`, `panel.json`) are the single source of truth — adding a model
or provider is a config edit, not a code change.

## 3. Subsystems

### 3.1 Routing & cross-provider failover (`scheduler.py`)
`SlotScheduler` owns rotation and resilience. `pick_slot_from(candidates, role,
exclude_providers)` returns the best eligible host for a logical model, where eligible =
not blacklisted, not in cooldown, not excluded. Ranking favors the least-recently-attempted
provider (rotation), then healthier model families, then freshness — so a flaky provider
drifts to the back instead of being hammered. `route_judge` (`jobs.py`) drives the bounce
loop: on any 429/5xx/empty/timeout it records a penalty (cooldown or blacklist), emits a
metric, and re-picks the next provider hosting the *same logical model*. Retries are
cross-provider by default (`JUDGE_RETRIES=0`): we move hosts rather than re-hit a failing
one. **Pacing** reserves each provider's next allowed call time under a lock
(`wait_for_provider_pacing`), so concurrent callers across both planes stagger to
distinct slots and stay under per-provider RPM.

### 3.2 Privacy router (`jobs.py` `_privacy_tiers` / `_pick_tiered`, `providers.slot_is_privacy_safe`)
Each provider carries a `trains_on_data` posture (`true`/`false`/`"per-model"`); a host is
privacy-safe iff it does not train on or log submissions. OpenRouter `:free` routes require
training consent, so they are `per-model` → unsafe; HF Inference is a proxy whose posture is
the routed backend's. Three request modes: **strict** (default) drops every unsafe host and
returns `privacy_blocked` rather than leak; **fallback** tries safe hosts first and an unsafe
host only as a last resort; **off** uses all. `PRIVACY_MODE` locks an instance to strict;
`no_store` runs a job without persisting its content anywhere.

### 3.3 Prompt cache (`promptcache.py`)
A per-space exact-match cache keyed by `sha256(logical, system_prompt, user_msg, max_tokens,
reasoning, research, privacy)`. **Privacy is part of the key** — a result produced under
`privacy=off` can never be served to a `strict` request. In-memory LRU (`CACHE_MAX=512`) +
atomic disk mirror, TTL `CACHE_TTL_S=24h`. Only successful calls are cached; `no_store`
skips writes. A hit returns prior output with zero provider calls and transmits nothing.

### 3.4 Durable async jobs (`jobs.py` `JobRunner`, `jobstore.py`)
Free containers recycle on deploy, which previously killed long async runs. `JobStore`
writes each job to `data/jobs/<id>.json` via atomic tmp-rename on every state transition and
mirrors to a private HF Dataset, so jobs survive both crashes and redeploys. On boot the
runner re-hydrates: panel jobs reschedule any unfinished judge (the `_exec` block carries the
prompt/budgets needed to resume), and mid-flight orchestration runs are marked `interrupted`.
`MAX_INFLIGHT_JOBS=200` caps simultaneously-running jobs (the async plane bypasses the HTTP
worker cap); `JOB_TTL_S=6h` prunes finished jobs.

### 3.5 Codebase ingestion (`repopack.py`)
Turns `{path: content}` or a fetched `repo_url` tarball (GitHub codeload, stdlib only,
`REPO_MAX_BYTES=50MB`) into one annotated blob: binaries/lockfiles/`node_modules` filtered,
per-file `===== FILE: =====` delimiters, token estimate chars/3 for code. Each judge gets a
pack **fitted to its own context window** (`ctx` per host in `models_catalog.json`); when a
repo exceeds a judge's budget, low-value files (generated/bulk) are dropped first, tests and
docs last, and the result reports `coverage` so partial-context reviews are honest. A
`diff_mode {base_ref, head_ref}` packs only the changed files plus a unified diff.

### 3.6 Orchestrator & template library (`orchestrator/`, `orchestrator/templates/`)
Beyond one-shot panels, `/api/run` executes multi-stage **schematics**: stages run
sequentially, a stage may **fan out** (one shard per item in a JSON list a prior stage
emitted, shards spread across providers via shared in-flight exclusion), and a **judge loop**
retries the final body up to `max_rounds` if rules/criteria fail. The built-in power
templates — `repo_audit`, `design_doc`, `redteam`, `panel_debate` — each map → fan-out →
synthesize → judge, and bake in hard-won lessons (e.g. `repo_audit` quarantines
confident-but-unverifiable concurrency claims into a "False-positive watch" section).
File-output mode reconstructs a multi-file artifact tree from marker blocks.

### 3.7 Three-layer judge (`judge/`)
Output hardening runs in order and merges reports: **Layer 1** declarative rules
(`engine.py`: min/max words, required sections, banned phrases, regex); **Layer 2** code
plugins (`citation_integrity`, `url_health`, `json_parses`); **Layer 3** LLM-as-judge
natural-language criteria. Hard fails trigger a retry round; soft flags annotate.

### 3.8 Per-profile metrics (`metrics.py`)
Every call emits a metric tagged with a `profile`, so cost/throttle/latency are tracked per
identity, not globally. Rolling per-provider call/token windows (pruned on write, 24h) feed a
**budget guard** that cools a provider when a profile hits its published daily ceiling.
`HOT_WINDOW=5000` events in memory; flushed every `FLUSH_EVERY=25` and mirrored to the HF
Dataset. Outputs are sampled only if `METRICS_SAMPLE_OUTPUTS=1` (off by default), and never
for `no_store` jobs. Exposed via `GET /api/stats`, `/api/metrics?profile=`, `/api/roster`.

## 4. Guarantees

- **≥2 privacy-safe frontier models, always.** `panel.roster()` joins the benchmark catalog
  with live host health and the privacy classification; `/health.frontier_ok` is true iff at
  least two *frontier* logical models have a privacy-safe host routable right now. It reports
  false honestly rather than silently routing to an unsafe or non-frontier host.
- **Privacy by isolation + verifiable routing.** One instance per user with their own keys;
  strict-by-default routing; `no_store` for zero retention. Backpressure responses (503/429)
  carry no prompt content.
- **Durability across crash and redeploy.** Atomic local writes + private HF Dataset mirror;
  reload-on-boot resumes unfinished panel judges.

## 5. Threat model & security

- **Auth.** Every non-health route is bearer-gated with a constant-time compare
  (`hmac.compare_digest`); an unset `CRITIQUE_TOKEN` refuses service (503) rather than
  serving open.
- **Resource exhaustion / DoS.** `BoundedThreadingHTTPServer` caps concurrent sync workers
  (`MAX_WORKERS=48`) and returns 503+`Retry-After` over capacity without spawning a thread;
  `REQUEST_TIMEOUT_S=30` drops slowloris connections; `MAX_INFLIGHT_JOBS=200` caps the async
  plane with 429+`Retry-After`; `MAX_BODY_BYTES=8MB` bounds request size. Per-user instances
  make these per-tenant by construction; the frontend can tune the env knobs per user.
- **Data at rest.** Job snapshots and the cache live on the instance's local disk and the
  user's *private* HF Dataset. `no_store` opts a job out entirely (trading deploy-resume for
  zero retention). Metrics store no prompt/output text unless explicitly enabled.
- **Provider trust.** The privacy router is the control that keeps prompts off
  training/logging hosts; `:free` and unpinned-proxy routes are classified unsafe.
- **Secret hygiene.** Keys are Space secrets, never committed; `data/` is gitignored.

## 6. Evaluation

- **Offline simulation (zero quota).** A `MOCK_MODE` provider yields deterministic,
  fault-injectable responses; nine sims exercise rotation/failover/budget/profile isolation
  (`sim_scenarios`), research ReAct (`sim_research`), crash-resume (`sim_persistence`),
  orchestration + fanout diversity (`sim_orchestrator`), the privacy router (`sim_privacy`),
  the cache incl. the privacy-key (`sim_cache`), ingestion + budget fitting (`sim_repopack`),
  the template library (`sim_templates`), and the concurrency bounds — 503/slowloris/429
  (`sim_server_limits`). All green.
- **Live verification.** Every roster slug is probed (16-token call) before catalog entry
  (`docs/ROSTER-2026-06.md`); failures are recorded and excluded, not shipped broken.
- **Whole-repo reality check.** A 163k-token review of this codebase confirmed strong hosts
  (Kimi, MiniMax) genuinely reason over the full repo, surfaced two real P1 bugs since fixed
  (cache privacy bypass; `privacy_blocked` fallthrough) and several false positives (later
  disproved), and exposed reliability variance (qwen3.5-397b returned empty content → demoted
  from the default panel).

## 7. Limitations & future work

- **`/api/run` does not context-fit the prompt.** The repo-pack per-judge budgeting lives in
  the `/api/panel` `files` form, not the orchestrator path — so a large prompt sent through a
  template hits small-context hosts (e.g. Cloudflare llama-3.3, 24k) with **413** and only
  survives via cross-provider failover to big-context hosts. Fix: have the scheduler skip
  hosts whose `ctx` cannot fit the estimated input, or extend ingestion budgeting to `/api/run`.
- **Per-request event loop on the sync plane.** Each sync request runs `asyncio.run` with a
  fresh `httpx.AsyncClient`, discarding connection pooling; acceptable under the worker cap,
  but a shared loop/client would be more efficient at scale.
- **Benchmarks are indicative.** `benchmarks.json` scores are hand-curated and dated, used
  only to order the roster and define "frontier" — not an authoritative ranking.
- **Provider breadth.** Current pool is four providers; expanding to Groq, Cerebras, Mistral
  La Plateforme, and HF Inference (each with a privacy block + live probe) widens the
  frontier-safety margin (planned).

## 8. Reproducibility

Duplicate the Hugging Face Space, set your own provider keys as Space secrets
(`NVIDIA_API_KEY`, `CF_API_TOKEN`+`CF_ACCOUNT_ID`, `GITHUB_TOKEN`, `OPENROUTER_API_KEY`) and
a `CRITIQUE_TOKEN`; optionally `METRICS_HF_REPO`/`JOBS_HF_REPO` for cross-deploy durability.
The service boots with whatever keys are present and reports its live posture at `/health`
and `/api/roster`. Everything is stdlib + `httpx`; no build step. See `README.md` for the
API surface and `PRIVACY.md` for the full privacy/off-switch matrix.