# Aurelius — Implementation Roadmap

This document records the codebase review that preceded the current round of
work and the phased plan that follows from it. It is written against the
post-rebuild architecture (`core/` + `adapters/` + `ingest/`, source-agnostic
`GraphNavigator`, SQLite `GraphStore`).

> **Update — optional Gemini layer.** An additive AI narration layer
> (`core/llm.py`) was later added: plain-English path explanations, finance
> Compare analysis, discover "why these" narratives, entity summaries, and a
> news coverage summary + refined tone. It is off unless `GEMINI_API_KEY` is
> set, is never in the search hot loop, and every AI endpoint degrades to a
> non-blocking notice (`reason:"no_key"|"rate_limited"|"cooling_down"`) with
> the standard non-LLM result always shown. See CLAUDE.md → "Optional Gemini
> layer."

---

## Part 1 — Review findings

### Architecture (sound, keep)
- **Adapter boundary holds.** The engine (`core/navigator.py`,
  `core/discovery.py`) speaks only `NodeRef`/`Edge` through the `GraphSource`
  protocol; Wikipedia, OpenAlex, GitHub are live adapters, Finance/News/Biomed
  answer from the store. Adding a domain is an adapter file plus (for
  ingested mode) an ingest script — no engine changes. This is exactly the
  property the Finance and News work below relies on, so it is reused as-is.
- **Search algorithm keeps its v1 lessons.** Greedy best-first with the
  depth tie-break, honest bidirectional meeting check (goal-zone d1/d2 with
  bridge bookkeeping), the `_real_edges`/`_validate_path` regression fence,
  stagnation escape, and `PRUNE_MIN_SURVIVORS`. No algorithm change needed;
  "bidirectional A*" in product terms maps to this meeting-check design
  (true A* is impossible here — `1 − cosine` is not admissible — and path
  optimality is not the product goal; a *meaningful, explainable* path is).
- **Store schema already supports typed, weighted edges**
  (`edges(source, src, dst, type, weight)`), so the richer finance graph and
  the news entity graph need zero schema migration for edges. Node kinds ride
  in the existing `features` JSON column.

### Bugs / hygiene found in review (fixed in Phase 0)
1. **`_ip_hits` rate-limit dict grows without bound** (`server.py`) — a
   `defaultdict(deque)` keyed by client IP is never pruned; empty deques for
   long-gone IPs accumulate for the life of the process. Fixed by sweeping
   stale entries during rate checks.
2. **Search timeout is silent** — when a WS search hits
   `MAX_SEARCH_SECONDS`, tasks are cancelled but the client is never told;
   the UI would sit on the last status line forever. Fixed by emitting a
   `not_found` timeout message.
3. **Superseded v1 modules still on disk** (`search.py`, `wiki.py`, root
   `embedding.py`) — no longer imported by anything but easy to edit by
   mistake. Deleted (git history keeps them).
4. **`data/*.db*` not git-ignored** — locally-generated SQLite (+ WAL/SHM)
   would land in commits. Added to `.gitignore`.
5. **No edge evidence for ingested sources** — `StoreBackedSource` inherited
   the no-op `edge_display`, so a finance path rendered with no indication of
   *why* consecutive nodes connect. Fixed with a typed, weighted display
   ("co-moves (corr 0.72)", "supplies", "holds") — this is the foundation of
   the "explainable paths" requirement.
6. **Unused import** (`random` in `server.py`) — removed.

### UI review (drives Phase 2)
- Nodes/links in the 3D view are oversized (`nodeRelSize 4`, sphere values up
  to 7) and the camera has no damping and no auto-fit, so a 100+ node search
  turns into one giant tangled ball you must fight to read.
- Every streamed node stays visible forever — no filtering, no
  level-of-detail, no way to collapse the exploration cloud once a path is
  found.
- The graph is display-only: you cannot expand a node to keep exploring after
  the search finishes.
- About / first-run copy still describes the Wikipedia-only v1.
- The Discover panel shows scored results with zero explanation of what a
  "hidden connection" is or why a given candidate appears.

---

## Part 2 — Phased plan

### Phase 0 — Fixes & hygiene (no behavior change) ✅
Everything in "Bugs / hygiene" above, plus this document.

### Phase 1 — Explorer backend ✅
- `GET /api/neighbors?source=&q=&limit=` — resolve a query, return outbound
  edges with `type`, `weight`, and human `display`, powering click-to-expand.
- `edge_display` for all store-backed sources (typed evidence).

### Phase 2 — 3D graph redesign ✅
- Smaller nodes (relSize ~2.2, tighter size ladder), thinner links, more
  inter-node spacing (weaker charge + longer link distance).
- Orbit controls with inertial damping; smooth `zoomToFit` after growth
  bursts; camera fly-to on focus (already present, retimed).
- Clutter control: view filters (**All / Explored / Path**), label
  level-of-detail (labels only on structurally important nodes; sprite size
  down), post-found dimming of the exploration cloud.
- Progressive expansion: click a node after the search settles → fetch its
  neighbors from `/api/neighbors` → animate them in (capped per expansion).
- Edge tooltips show the relationship type + weight (evidence surfacing).

### Phase 3 — Hidden Connections explainer + modals ✅
- Discover panel opens with a plain-language explanation of what hidden
  connections are (indirect links through shared neighbors, semantic
  similarity without a direct edge, co-movement, shared dependencies…),
  and each result carries a "why you're seeing this" line built from its
  actual bridges and similarity.
- About + first-run hero copy rewritten for the multi-domain engine.

### Phase 4 — Finance graph, typed ✅
Ingest builds a heterogeneous graph (node kinds ride in `features.kind`):
- **Companies** (Yahoo chart API: real prices, names) — as before.
- **Sectors** as first-class nodes; `sector_member` replaces the old
  O(n²) pairwise `same_sector` edges.
- **ETFs** (SPY, QQQ, DIA, sector SPDRs) with `holds` edges to constituents.
- **Executives** (`led_by`), **countries** (`based_in`) from a curated map.
- **Supply chain** (`supplies`) and **competitors** (`competes_with`) from a
  curated, sourced seed set — the classic pairs (TSM→AAPL/NVDA/AMD, V–MA,
  F–GM–TSLA…).
- **Ownership** (`owns_stake`): Berkshire Hathaway's flagship stakes.
- **Macro indicators** (10-y yield, WTI crude, dollar index, gold, S&P 500,
  VIX) fetched like any ticker; `macro_correlates` edges are *computed* from
  return correlation, so the macro linkage is evidence, not assertion.
- `co_moves` correlation kNN edges kept (the structural workhorse).
Every edge type has a human display string → every path step is explainable.
Deferred (documented, not blocking): timeline mode, live 13F ownership
ingestion, event-propagation explorer (needs the news graph first — see
Phase 5 wiring).

### Phase 5 — News Intelligence subsystem ✅
A standalone package `news_intel/`, deliberately not finance-specific:
- **Connectors** (`connectors.py`): `NewsConnector` interface; NewsAPI +
  GDELT implemented; RSS/Reddit/SEC/etc. are new connector classes, nothing
  else changes.
- **Store** (`store.py`): raw `news_articles` table (URL-hash dedup,
  re-fetch updates), entity link table, story tables — same SQLite file,
  own namespace.
- **Pipeline** (`pipeline.py`): Fetch → Clean(dedup) → Extract(entities) →
  Embed → Build relationships → Update graph → Index, each stage a function.
- **Stories** (`stories.py`): embedding + shared-entity clustering groups
  articles into evolving stories.
- **Ranking** (`rank.py`): relevance × freshness decay × source credibility ×
  entity graph-degree.
- **Service** (`service.py`) + REST: `/api/news/search`, `/api/news/stories`,
  `/api/news/entity`, `/api/news/status`, `/api/news/refresh` — any domain
  module (Finance, Biology…) consumes these; nothing in the package imports
  from a domain.
- Optional scheduled refresh (env `NEWS_REFRESH_MINUTES`, off by default).
- Graph output lands in the existing `news` source (entities + article
  nodes + `mentions`/`co_mentioned` edges), so navigator/discover/3D UI work
  on it with zero changes — the proof of loose coupling.

### Phase 6 — Verify, document, ship ✅
End-to-end verification in the live preview; CLAUDE.md / README /
AURELIUS_LEARNINGS updated; deploy steps for Hugging Face Spaces (backend)
and Vercel (frontend).

### Future (deliberately out of scope now)
- Postgres + pgvector swap (`GraphStore` interface is already shaped for it).
- GraphSAGE upgrade over node2vec; cross-source entity resolution
  (news "Nvidia" ↔ finance NVDA ↔ wikipedia "Nvidia").
- Timeline mode over story clusters; event-propagation explorer
  (news event → entities → finance exposure).
- LLM explanation layer (optional service, explicitly outside the core).