| # AGENTS.md |
|
|
| Agent entry point for WorldMonitor. Read this first, then follow links for depth. |
|
|
| ## What This Project Is |
|
|
| Real-time global intelligence dashboard. TypeScript SPA (Vite + Preact) with 179 top-level TypeScript component files, 80+ Vercel Edge API endpoint entries, a Tauri desktop app with Node.js sidecar, and a Railway relay service. Aggregates geopolitics, military, finance, climate, cyber, maritime, and aviation data across 35 freshness-tracked source groups. |
|
|
| ## Repository Map |
|
|
| ``` |
| . |
| ├── src/ # Browser SPA (TypeScript, class-based components) |
| │ ├── app/ # App orchestration (data-loader, refresh-scheduler, panel-layout) |
| │ ├── bootstrap/ # Startup/recovery (chunk reload, deferred Sentry, SW update) |
| │ ├── components/ # 179 top-level TypeScript component files |
| │ ├── config/ # Variant configs, panel/layer definitions, market symbols |
| │ ├── services/ # Business logic (218 service modules and domain directories) |
| │ ├── shared/ # Cross-cutting helpers (premium paths, registries, staleness) |
| │ ├── embed/ # Embeddable widget loader |
| │ ├── styles/ # Global CSS (layers, themes, panel styles) |
| │ ├── shims/ # Runtime shims (child-process for sidecar) |
| │ ├── data/ # Static JSON datasets (conservation, renewable, happiness) |
| │ ├── e2e/ # Map test harnesses (consumed by Playwright specs) |
| │ ├── types/ # TypeScript type definitions |
| │ ├── utils/ # Shared utilities (circuit-breaker, theme, URL state, DOM) |
| │ ├── workers/ # Web Workers (analysis, ML/ONNX, vector DB) |
| │ ├── generated/ # Proto-generated client/server stubs (DO NOT EDIT) |
| │ ├── locales/ # i18n translation files |
| │ └── App.ts # Main application entry |
| ├── api/ # Vercel Edge Functions (plain JS, self-contained) |
| │ ├── _*.js # Shared helpers (CORS, rate-limit, API key, relay) |
| │ ├── health.js # Health check endpoint |
| │ ├── bootstrap.js # Bulk data hydration endpoint |
| │ └── <domain>/ # Domain-specific endpoints (aviation/, climate/, etc.) |
| ├── server/ # Server-side shared code (used by Edge Functions) |
| │ ├── _shared/ # Redis, rate-limit, LLM, caching, response headers |
| │ ├── gateway.ts # Domain gateway factory (CORS, auth, cache tiers) |
| │ ├── router.ts # Route matching |
| │ └── worldmonitor/ # Domain handlers (mirrors proto service structure) |
| ├── proto/ # Protobuf definitions (sebuf framework) |
| │ ├── buf.yaml # Buf configuration |
| │ └── worldmonitor/ # Service definitions with HTTP annotations |
| ├── shared/ # Cross-platform data (JSON configs for markets, RSS domains) |
| ├── data/ # Static data (telegram channels, OREF threat translations, gamma irradiators) |
| ├── public/ # Static assets served as-is (favicons, textures, .well-known, llms.txt) |
| ├── scripts/ # Seed scripts, build helpers, data fetchers |
| ├── src-tauri/ # Tauri desktop shell (Rust + Node.js sidecar) |
| │ └── sidecar/ # Node.js sidecar API server |
| ├── consumer-prices-core/ # Consumer-price scrapers (Playwright, per-country baskets; Railway/Docker) |
| ├── workers/ # Cloudflare Workers (edge CORS preflight for api.worldmonitor.app) |
| ├── tests/ # Unit/integration tests (node:test runner) |
| ├── e2e/ # Playwright E2E specs |
| ├── pro-test/ # Standalone Pro QA app (separate package) |
| ├── docs/ # Mintlify documentation site |
| │ └── solutions/ # Documented solutions to past problems (bugs, patterns, practices) — YAML frontmatter (module, tags, problem_type) |
| ├── docker/ # Docker build for Railway services |
| ├── deploy/ # Deployment configs (nginx) |
| ├── CONCEPTS.md # Shared domain vocabulary (entities, named processes, status concepts) |
| └── blog-site/ # Static blog (built into public/blog/) |
| ``` |
|
|
| ## How to Run |
|
|
| ```bash |
| npm ci # Deterministic install (also runs blog-site postinstall) |
| npm run dev # Start Vite dev server (full variant) |
| npm run dev:tech # Start tech-only variant |
| npm run dev:energy # Start energy-security variant |
| npm run typecheck # tsc --noEmit (strict mode) |
| npm run typecheck:api # Typecheck API layer separately |
| npm run test:data # Run unit/integration tests |
| npm run test:sidecar # Run sidecar + API handler tests |
| npm run test:e2e # Run all Playwright E2E tests |
| make generate # Regenerate proto stubs + per-service & unified OpenAPI specs (requires buf + sebuf v0.11.1 plugins) |
| npm run worktree:bootstrap # Fresh worktree: link local env files + npm ci with tmp cache |
| npm run worktree:bootstrap:test-only # Fresh docs/test worktree: same, but npm ci --ignore-scripts |
| npm run worktree:env # Link ignored local env files only |
| ``` |
|
|
| ## Fresh Worktree Bootstrap |
|
|
| Worktrees usually start without ignored local state. When creating or entering one: |
|
|
| 1. Start from `origin/main` or the requested base, not a dirty local branch. |
| 2. Run `npm run worktree:bootstrap` before typecheck/tests. The helper links ignored `.env.local` / `.env` from the main worktree when Git can infer it, and installs deps with `npm ci --cache /tmp/worldmonitor-npm-cache`. |
| 3. If only docs/test tooling is needed and native postinstall work is unnecessary, use `npm run worktree:bootstrap:test-only`. |
| 4. If live credentials are unavailable, do not fabricate secrets. Run the non-credentialed checks you can and report the credential gate explicitly. |
|
|
| Env rules: |
|
|
| - Link only `.env.local` and `.env`. Never copy or link `.env.vercel-backup` or `.env.vercel-export`; the pre-push guard blocks those files even as symlinks. |
| - Override env source discovery with `WM_ENV_SOURCE=/path/to/worldmonitor npm run worktree:env` when the main worktree cannot be inferred. |
| - `.env*` files are ignored local state. Do not add, print, or summarize secret values. |
|
|
| Validation hygiene: |
|
|
| - Prefer `npm ci` over `npm install` in fresh worktrees. Use `npm_config_cache=/tmp/worldmonitor-npm-cache` for `npx` or install commands if cache ownership errors appear. |
| - After bootstrap or pre-push, run `git status --short`. If dependency bootstrap changed lockfiles you did not intend to edit, remove those incidental changes before finalizing. |
| - After install, prefer local tools such as `./node_modules/.bin/tsx --test ...` for focused TypeScript tests when `npx` is flaky. |
|
|
| ## Architecture Rules |
|
|
| ### Dependency Direction |
|
|
| ``` |
| types -> config -> services -> components -> app -> App.ts |
| ``` |
|
|
| - `types/` has zero internal imports |
| - `config/` imports only from `types/` |
| - `services/` imports from `types/` and `config/` |
| - `components/` imports from all above |
| - `app/` orchestrates components and services |
|
|
| ### API Layer Constraints |
|
|
| - `api/*.js` are Vercel Edge Functions: **self-contained JS only** |
| - They CANNOT import from `../src/` or `../server/` (different runtime) |
| - Only same-directory `_*.js` helpers and npm packages |
| - Enforced by `tests/edge-functions.test.mjs` and pre-push hook esbuild check |
|
|
| ### Server Layer |
|
|
| - `server/` code is bundled INTO Edge Functions at deploy time via gateway |
| - `server/_shared/` contains Redis client, rate limiting, LLM helpers |
| - `server/worldmonitor/<domain>/` has RPC handlers matching proto services |
| - All handlers use `cachedFetchJson()` for Redis caching with stampede protection |
|
|
| ### Proto Contract Flow |
|
|
| ``` |
| proto/ definitions -> buf generate -> src/generated/{client,server}/ -> handlers wire up |
| ``` |
|
|
| - GET fields need `(sebuf.http.query)` annotation |
| - `repeated string` fields need `parseStringArray()` in handler |
| - `int64` maps to `string` in TypeScript |
| - CI checks proto freshness via `.github/workflows/proto-check.yml` |
|
|
| ## Variant System |
|
|
| The app ships multiple variants with different panel/layer configurations: |
|
|
| - `full` (default): All features |
| - `tech`: Technology-focused subset |
| - `finance`: Financial markets focus |
| - `commodity`: Commodity markets focus |
| - `happy`: Positive news only |
| - `energy`: Energy security, chokepoints, oil/gas, and disruption timelines |
|
|
| Variant is set via `VITE_VARIANT` env var. Config lives in `src/config/variants/`. |
|
|
| ## Key Patterns |
|
|
| ### Adding a New API Endpoint |
|
|
| 1. Define proto message in `proto/worldmonitor/<domain>/` |
| 2. Add RPC with `(sebuf.http.config)` annotation |
| 3. Run `make generate` |
| 4. Create handler in `server/worldmonitor/<domain>/` |
| 5. Wire handler in domain's `handler.ts` |
| 6. Use `cachedFetchJson()` for caching, include request params in cache key |
|
|
| ### Adding a New Panel |
|
|
| 1. Create `src/components/MyPanel.ts` extending `Panel` |
| 2. Register in `src/config/panels.ts` |
| 3. Add to variant configs in `src/config/variants/` |
| 4. Wire data loading in `src/app/data-loader.ts` |
|
|
| ### Circuit Breakers |
|
|
| - `src/utils/circuit-breaker.ts` for client-side |
| - Used in data loaders to prevent cascade failures |
| - Separate breaker per data domain |
|
|
| ### Caching |
|
|
| - Redis (Upstash) via `server/_shared/redis.ts` |
| - `cachedFetchJson()` coalesces concurrent cache misses |
| - Cache tiers: fast (5m), medium (10m), slow (30m), static (2h), daily (24h) |
| - Cache key MUST include request-varying params |
|
|
| ## Testing |
|
|
| - **Unit/Integration**: `tests/*.test.{mjs,mts}` using `node:test` runner |
| - **Sidecar tests**: `api/*.test.mjs`, `src-tauri/sidecar/*.test.mjs` |
| - **E2E**: `e2e/*.spec.ts` using Playwright |
| - **Visual regression**: Golden screenshot comparison per variant |
|
|
| ## CI Checks (GitHub Actions) |
|
|
| | Workflow | Trigger | What it checks | |
| |---|---|---| |
| | `typecheck.yml` | PR + push to main | `tsc --noEmit` for src and API | |
| | `lint.yml` | PR (markdown changes) | markdownlint-cli2 | |
| | `proto-check.yml` | PR (proto changes) | Generated code freshness | |
| | `build-desktop.yml` | Manual | Tauri desktop build | |
| | `test-linux-app.yml` | Manual | Linux AppImage smoke test | |
|
|
| ## Pre-Push Hook |
|
|
| Runs automatically before `git push`. Two tiers: |
|
|
| **Always (state-dependent, fast — run even on a cache hit):** local Vercel env-dump guard, PR-state check (no pushes to merged/closed PR branches), branch-contamination guard (>20 commits ahead), `scripts/` lockfile sync. |
|
|
| **Tree-dependent (skipped entirely on a green-tree cache hit):** Unicode safety and version sync (always run for uncached trees), plus the diff-scoped checks: TypeScript (frontend tsc on `src/`-surface changes; `typecheck:api` on `api/|server/|scripts/|src/generated/`; Convex tsc on `convex/`), CJS syntax, boundary/safe-html/Sentry-coverage/rate-limit/premium-fetch lints (each also fires when its own guardrail script changes), edge esbuild check (`api/|server/|src/generated/` — edge entries bundle-import server code), markdown/MDX lint, proto + pro-test bundle freshness, change-scoped tests. `package.json`/`tsconfig` changes — or an unresolvable `origin/main` diff — force everything (an unresolvable diff also bypasses the green-tree cache: a blind run trusts nothing, including prior attestations). |
|
|
| **Green-tree cache:** a tree that passed the full gate is recorded (`$GIT_DIR/wm-prepush-green`); re-pushing the identical tree (remote failure, message-only amend) skips all tree-dependent checks — same tree, same result. Delete that file to force a full re-run. |
|
|
| Heavy checks (`test:data`, typechecks, edge-bundle) must run **sequentially** in worktrees — parallel runs OOM (exit 137). |
|
|
| ## Shipping Velocity (Agent Workflow) |
|
|
| - **Before starting work on an issue:** check for parallel/duplicate work first — `gh pr list --search "<issue#>"` AND `git worktree list` (background codex/claude sessions ship PRs under the same account). |
| - **PR delivery authority:** a user request to implement, fix, or ship a scoped change authorizes creating and updating the ready PRs needed to deliver it, including corrective follow-up PRs discovered by review or CI, plus monitoring and repairing those PRs without additional per-PR confirmation. This authority is limited to the requested change and its delivery branches; review-only or diagnostic requests remain read-only. |
| - **Merge authority is explicit and non-delegable:** never merge a PR, enable auto-merge, queue a merge, or run any equivalent GitHub merge action unless the user has explicitly requested that specific action in the current conversation. A request to implement, ship, push, create a PR, or monitor CI does **not** authorize merging. Wait for clear approval and report the ready state instead. |
| - **After pushing a PR:** do not sleep-poll CI. Start `gh pr checks <n> --watch` as a background task, or report the current check state; never turn on auto-merge without the explicit approval above. |
| - **docs/plans/ is gitignored** — plan documents are local working state and do not travel between worktrees or ship in PRs. |
| - **PR-review verification:** never assert a finding is fixed/stale from memory — re-fetch the PR head SHA and diff the cited lines first. |
|
|
| ## Deployment |
|
|
| - **Web**: Vercel (auto-deploy on push to main) |
| - **Relay/Seeds**: Railway (Docker, cron services) |
| - **Desktop**: Tauri builds via GitHub Actions |
| - **Docs**: Mintlify (proxied through Vercel at `/docs`) |
|
|
| ## Critical Conventions |
|
|
| - `fetch.bind(globalThis)` is BANNED. Use `(...args) => globalThis.fetch(...args)` instead |
| - Edge Functions cannot use `node:http`, `node:https`, `node:zlib` |
| - Always include `User-Agent` header in server-side fetch calls |
| - Yahoo Finance requests must be staggered (150ms delays) |
| - New data sources MUST have bootstrap hydration wired in `api/bootstrap.js` |
| - Redis seed scripts MUST write `seed-meta:<key>` for health monitoring |
| - Seed credentials load only via `loadEnvFile()` (inert under test runtimes, resolves `.env.local` at the checkout root, `only:` narrows the keys) — never hand-roll a `.env` reader or resolve one from `$HOME` or an absolute literal. Note `worktree:bootstrap` symlinks the source checkout's `.env.local`, so a bootstrapped worktree shares real credentials when a seeder is actually run |
|
|
| ## External References |
|
|
| - [Architecture (system reference)](ARCHITECTURE.md) |
| - [Design Philosophy (why decisions were made)](docs/architecture.mdx) |
| - [Contributing guide](CONTRIBUTING.md) |
| - [Data sources catalog](docs/data-sources.mdx) |
| - [Health endpoints](docs/health-endpoints.mdx) |
| - [Adding endpoints guide](docs/adding-endpoints.mdx) |
| - [API reference (OpenAPI)](docs/api/) |
|
|
