| # Aurelius — Implementation Roadmap |
|
|
| This document records the codebase review that preceded the current round of |
| work and the phased plan that follows from it. It is written against the |
| post-rebuild architecture (`core/` + `adapters/` + `ingest/`, source-agnostic |
| `GraphNavigator`, SQLite `GraphStore`). |
|
|
| > **Update — optional Gemini layer.** An additive AI narration layer |
| > (`core/llm.py`) was later added: plain-English path explanations, finance |
| > Compare analysis, discover "why these" narratives, entity summaries, and a |
| > news coverage summary + refined tone. It is off unless `GEMINI_API_KEY` is |
| > set, is never in the search hot loop, and every AI endpoint degrades to a |
| > non-blocking notice (`reason:"no_key"|"rate_limited"|"cooling_down"`) with |
| > the standard non-LLM result always shown. See CLAUDE.md → "Optional Gemini |
| > layer." |
| |
| --- |
| |
| ## Part 1 — Review findings |
| |
| ### Architecture (sound, keep) |
| - **Adapter boundary holds.** The engine (`core/navigator.py`, |
| `core/discovery.py`) speaks only `NodeRef`/`Edge` through the `GraphSource` |
| protocol; Wikipedia, OpenAlex, GitHub are live adapters, Finance/News/Biomed |
| answer from the store. Adding a domain is an adapter file plus (for |
| ingested mode) an ingest script — no engine changes. This is exactly the |
| property the Finance and News work below relies on, so it is reused as-is. |
| - **Search algorithm keeps its v1 lessons.** Greedy best-first with the |
| depth tie-break, honest bidirectional meeting check (goal-zone d1/d2 with |
| bridge bookkeeping), the `_real_edges`/`_validate_path` regression fence, |
| stagnation escape, and `PRUNE_MIN_SURVIVORS`. No algorithm change needed; |
| "bidirectional A*" in product terms maps to this meeting-check design |
| (true A* is impossible here — `1 − cosine` is not admissible — and path |
| optimality is not the product goal; a *meaningful, explainable* path is). |
| - **Store schema already supports typed, weighted edges** |
| (`edges(source, src, dst, type, weight)`), so the richer finance graph and |
| the news entity graph need zero schema migration for edges. Node kinds ride |
| in the existing `features` JSON column. |
| |
| ### Bugs / hygiene found in review (fixed in Phase 0) |
| 1. **`_ip_hits` rate-limit dict grows without bound** (`server.py`) — a |
| `defaultdict(deque)` keyed by client IP is never pruned; empty deques for |
| long-gone IPs accumulate for the life of the process. Fixed by sweeping |
| stale entries during rate checks. |
| 2. **Search timeout is silent** — when a WS search hits |
| `MAX_SEARCH_SECONDS`, tasks are cancelled but the client is never told; |
| the UI would sit on the last status line forever. Fixed by emitting a |
| `not_found` timeout message. |
| 3. **Superseded v1 modules still on disk** (`search.py`, `wiki.py`, root |
| `embedding.py`) — no longer imported by anything but easy to edit by |
| mistake. Deleted (git history keeps them). |
| 4. **`data/*.db*` not git-ignored** — locally-generated SQLite (+ WAL/SHM) |
| would land in commits. Added to `.gitignore`. |
| 5. **No edge evidence for ingested sources** — `StoreBackedSource` inherited |
| the no-op `edge_display`, so a finance path rendered with no indication of |
| *why* consecutive nodes connect. Fixed with a typed, weighted display |
| ("co-moves (corr 0.72)", "supplies", "holds") — this is the foundation of |
| the "explainable paths" requirement. |
| 6. **Unused import** (`random` in `server.py`) — removed. |
|
|
| ### UI review (drives Phase 2) |
| - Nodes/links in the 3D view are oversized (`nodeRelSize 4`, sphere values up |
| to 7) and the camera has no damping and no auto-fit, so a 100+ node search |
| turns into one giant tangled ball you must fight to read. |
| - Every streamed node stays visible forever — no filtering, no |
| level-of-detail, no way to collapse the exploration cloud once a path is |
| found. |
| - The graph is display-only: you cannot expand a node to keep exploring after |
| the search finishes. |
| - About / first-run copy still describes the Wikipedia-only v1. |
| - The Discover panel shows scored results with zero explanation of what a |
| "hidden connection" is or why a given candidate appears. |
|
|
| --- |
|
|
| ## Part 2 — Phased plan |
|
|
| ### Phase 0 — Fixes & hygiene (no behavior change) ✅ |
| Everything in "Bugs / hygiene" above, plus this document. |
|
|
| ### Phase 1 — Explorer backend ✅ |
| - `GET /api/neighbors?source=&q=&limit=` — resolve a query, return outbound |
| edges with `type`, `weight`, and human `display`, powering click-to-expand. |
| - `edge_display` for all store-backed sources (typed evidence). |
|
|
| ### Phase 2 — 3D graph redesign ✅ |
| - Smaller nodes (relSize ~2.2, tighter size ladder), thinner links, more |
| inter-node spacing (weaker charge + longer link distance). |
| - Orbit controls with inertial damping; smooth `zoomToFit` after growth |
| bursts; camera fly-to on focus (already present, retimed). |
| - Clutter control: view filters (**All / Explored / Path**), label |
| level-of-detail (labels only on structurally important nodes; sprite size |
| down), post-found dimming of the exploration cloud. |
| - Progressive expansion: click a node after the search settles → fetch its |
| neighbors from `/api/neighbors` → animate them in (capped per expansion). |
| - Edge tooltips show the relationship type + weight (evidence surfacing). |
|
|
| ### Phase 3 — Hidden Connections explainer + modals ✅ |
| - Discover panel opens with a plain-language explanation of what hidden |
| connections are (indirect links through shared neighbors, semantic |
| similarity without a direct edge, co-movement, shared dependencies…), |
| and each result carries a "why you're seeing this" line built from its |
| actual bridges and similarity. |
| - About + first-run hero copy rewritten for the multi-domain engine. |
|
|
| ### Phase 4 — Finance graph, typed ✅ |
| Ingest builds a heterogeneous graph (node kinds ride in `features.kind`): |
| - **Companies** (Yahoo chart API: real prices, names) — as before. |
| - **Sectors** as first-class nodes; `sector_member` replaces the old |
| O(n²) pairwise `same_sector` edges. |
| - **ETFs** (SPY, QQQ, DIA, sector SPDRs) with `holds` edges to constituents. |
| - **Executives** (`led_by`), **countries** (`based_in`) from a curated map. |
| - **Supply chain** (`supplies`) and **competitors** (`competes_with`) from a |
| curated, sourced seed set — the classic pairs (TSM→AAPL/NVDA/AMD, V–MA, |
| F–GM–TSLA…). |
| - **Ownership** (`owns_stake`): Berkshire Hathaway's flagship stakes. |
| - **Macro indicators** (10-y yield, WTI crude, dollar index, gold, S&P 500, |
| VIX) fetched like any ticker; `macro_correlates` edges are *computed* from |
| return correlation, so the macro linkage is evidence, not assertion. |
| - `co_moves` correlation kNN edges kept (the structural workhorse). |
| Every edge type has a human display string → every path step is explainable. |
| Deferred (documented, not blocking): timeline mode, live 13F ownership |
| ingestion, event-propagation explorer (needs the news graph first — see |
| Phase 5 wiring). |
|
|
| ### Phase 5 — News Intelligence subsystem ✅ |
| A standalone package `news_intel/`, deliberately not finance-specific: |
| - **Connectors** (`connectors.py`): `NewsConnector` interface; NewsAPI + |
| GDELT implemented; RSS/Reddit/SEC/etc. are new connector classes, nothing |
| else changes. |
| - **Store** (`store.py`): raw `news_articles` table (URL-hash dedup, |
| re-fetch updates), entity link table, story tables — same SQLite file, |
| own namespace. |
| - **Pipeline** (`pipeline.py`): Fetch → Clean(dedup) → Extract(entities) → |
| Embed → Build relationships → Update graph → Index, each stage a function. |
| - **Stories** (`stories.py`): embedding + shared-entity clustering groups |
| articles into evolving stories. |
| - **Ranking** (`rank.py`): relevance × freshness decay × source credibility × |
| entity graph-degree. |
| - **Service** (`service.py`) + REST: `/api/news/search`, `/api/news/stories`, |
| `/api/news/entity`, `/api/news/status`, `/api/news/refresh` — any domain |
| module (Finance, Biology…) consumes these; nothing in the package imports |
| from a domain. |
| - Optional scheduled refresh (env `NEWS_REFRESH_MINUTES`, off by default). |
| - Graph output lands in the existing `news` source (entities + article |
| nodes + `mentions`/`co_mentioned` edges), so navigator/discover/3D UI work |
| on it with zero changes — the proof of loose coupling. |
|
|
| ### Phase 6 — Verify, document, ship ✅ |
| End-to-end verification in the live preview; CLAUDE.md / README / |
| AURELIUS_LEARNINGS updated; deploy steps for Hugging Face Spaces (backend) |
| and Vercel (frontend). |
| |
| ### Future (deliberately out of scope now) |
| - Postgres + pgvector swap (`GraphStore` interface is already shaped for it). |
| - GraphSAGE upgrade over node2vec; cross-source entity resolution |
| (news "Nvidia" ↔ finance NVDA ↔ wikipedia "Nvidia"). |
| - Timeline mode over story clusters; event-propagation explorer |
| (news event → entities → finance exposure). |
| - LLM explanation layer (optional service, explicitly outside the core). |
| |
