ROADMAP.md

Aurelius — Implementation Roadmap

This document records the codebase review that preceded the current round of work and the phased plan that follows from it. It is written against the post-rebuild architecture (core/ + adapters/ + ingest/, source-agnostic GraphNavigator, SQLite GraphStore).

Update — optional Gemini layer. An additive AI narration layer (core/llm.py) was later added: plain-English path explanations, finance Compare analysis, discover "why these" narratives, entity summaries, and a news coverage summary + refined tone. It is off unless GEMINI_API_KEY is set, is never in the search hot loop, and every AI endpoint degrades to a non-blocking notice (reason:"no_key"|"rate_limited"|"cooling_down") with the standard non-LLM result always shown. See CLAUDE.md → "Optional Gemini layer."

---

Part 1 — Review findings

Architecture (sound, keep)

• Adapter boundary holds. The engine (core/navigator.py, core/discovery.py) speaks only NodeRef/Edge through the GraphSource protocol; Wikipedia, OpenAlex, GitHub are live adapters, Finance/News/Biomed answer from the store. Adding a domain is an adapter file plus (for ingested mode) an ingest script — no engine changes. This is exactly the property the Finance and News work below relies on, so it is reused as-is.
• Search algorithm keeps its v1 lessons. Greedy best-first with the depth tie-break, honest bidirectional meeting check (goal-zone d1/d2 with bridge bookkeeping), the _real_edges/_validate_path regression fence, stagnation escape, and PRUNE_MIN_SURVIVORS. No algorithm change needed; "bidirectional A*" in product terms maps to this meeting-check design (true A* is impossible here — 1 − cosine is not admissible — and path optimality is not the product goal; a meaningful, explainable path is).
• Store schema already supports typed, weighted edges (edges(source, src, dst, type, weight)), so the richer finance graph and the news entity graph need zero schema migration for edges. Node kinds ride in the existing features JSON column.

Bugs / hygiene found in review (fixed in Phase 0)

1. _ip_hits rate-limit dict grows without bound (server.py) — a defaultdict(deque) keyed by client IP is never pruned; empty deques for long-gone IPs accumulate for the life of the process. Fixed by sweeping stale entries during rate checks.
2. Search timeout is silent — when a WS search hits MAX_SEARCH_SECONDS, tasks are cancelled but the client is never told; the UI would sit on the last status line forever. Fixed by emitting a not_found timeout message.
3. Superseded v1 modules still on disk (search.py, wiki.py, root embedding.py) — no longer imported by anything but easy to edit by mistake. Deleted (git history keeps them).
4. data/*.db* not git-ignored — locally-generated SQLite (+ WAL/SHM) would land in commits. Added to .gitignore.
5. No edge evidence for ingested sources — StoreBackedSource inherited the no-op edge_display, so a finance path rendered with no indication of why consecutive nodes connect. Fixed with a typed, weighted display ("co-moves (corr 0.72)", "supplies", "holds") — this is the foundation of the "explainable paths" requirement.
6. Unused import (random in server.py) — removed.

UI review (drives Phase 2)

• Nodes/links in the 3D view are oversized (nodeRelSize 4, sphere values up to 7) and the camera has no damping and no auto-fit, so a 100+ node search turns into one giant tangled ball you must fight to read.
• Every streamed node stays visible forever — no filtering, no level-of-detail, no way to collapse the exploration cloud once a path is found.
• The graph is display-only: you cannot expand a node to keep exploring after the search finishes.
• About / first-run copy still describes the Wikipedia-only v1.
• The Discover panel shows scored results with zero explanation of what a "hidden connection" is or why a given candidate appears.

---

Part 2 — Phased plan

Phase 0 — Fixes & hygiene (no behavior change) ✅

Everything in "Bugs / hygiene" above, plus this document.

Phase 1 — Explorer backend ✅

• GET /api/neighbors?source=&q=&limit= — resolve a query, return outbound edges with type, weight, and human display, powering click-to-expand.
• edge_display for all store-backed sources (typed evidence).

Phase 2 — 3D graph redesign ✅

• Smaller nodes (relSize ~2.2, tighter size ladder), thinner links, more inter-node spacing (weaker charge + longer link distance).
• Orbit controls with inertial damping; smooth zoomToFit after growth bursts; camera fly-to on focus (already present, retimed).
• Clutter control: view filters (All / Explored / Path), label level-of-detail (labels only on structurally important nodes; sprite size down), post-found dimming of the exploration cloud.
• Progressive expansion: click a node after the search settles → fetch its neighbors from /api/neighbors → animate them in (capped per expansion).
• Edge tooltips show the relationship type + weight (evidence surfacing).

Phase 3 — Hidden Connections explainer + modals ✅

• Discover panel opens with a plain-language explanation of what hidden connections are (indirect links through shared neighbors, semantic similarity without a direct edge, co-movement, shared dependencies…), and each result carries a "why you're seeing this" line built from its actual bridges and similarity.
• About + first-run hero copy rewritten for the multi-domain engine.

Phase 4 — Finance graph, typed ✅

Ingest builds a heterogeneous graph (node kinds ride in features.kind):

• Companies (Yahoo chart API: real prices, names) — as before.
• Sectors as first-class nodes; sector_member replaces the old O(n²) pairwise same_sector edges.
• ETFs (SPY, QQQ, DIA, sector SPDRs) with holds edges to constituents.
• Executives (led_by), countries (based_in) from a curated map.
• Supply chain (supplies) and competitors (competes_with) from a curated, sourced seed set — the classic pairs (TSM→AAPL/NVDA/AMD, V–MA, F–GM–TSLA…).
• Ownership (owns_stake): Berkshire Hathaway's flagship stakes.
• Macro indicators (10-y yield, WTI crude, dollar index, gold, S&P 500, VIX) fetched like any ticker; macro_correlates edges are computed from return correlation, so the macro linkage is evidence, not assertion.
• co_moves correlation kNN edges kept (the structural workhorse). Every edge type has a human display string → every path step is explainable. Deferred (documented, not blocking): timeline mode, live 13F ownership ingestion, event-propagation explorer (needs the news graph first — see Phase 5 wiring).

Phase 5 — News Intelligence subsystem ✅

A standalone package news_intel/, deliberately not finance-specific:

• Connectors (connectors.py): NewsConnector interface; NewsAPI + GDELT implemented; RSS/Reddit/SEC/etc. are new connector classes, nothing else changes.
• Store (store.py): raw news_articles table (URL-hash dedup, re-fetch updates), entity link table, story tables — same SQLite file, own namespace.
• Pipeline (pipeline.py): Fetch → Clean(dedup) → Extract(entities) → Embed → Build relationships → Update graph → Index, each stage a function.
• Stories (stories.py): embedding + shared-entity clustering groups articles into evolving stories.
• Ranking (rank.py): relevance × freshness decay × source credibility × entity graph-degree.
• Service (service.py) + REST: /api/news/search, /api/news/stories, /api/news/entity, /api/news/status, /api/news/refresh — any domain module (Finance, Biology…) consumes these; nothing in the package imports from a domain.
• Optional scheduled refresh (env NEWS_REFRESH_MINUTES, off by default).
• Graph output lands in the existing news source (entities + article nodes + mentions/co_mentioned edges), so navigator/discover/3D UI work on it with zero changes — the proof of loose coupling.

Phase 6 — Verify, document, ship ✅

End-to-end verification in the live preview; CLAUDE.md / README / AURELIUS_LEARNINGS updated; deploy steps for Hugging Face Spaces (backend) and Vercel (frontend).

Future (deliberately out of scope now)

• Postgres + pgvector swap (GraphStore interface is already shaped for it).
• GraphSAGE upgrade over node2vec; cross-source entity resolution (news "Nvidia" ↔ finance NVDA ↔ wikipedia "Nvidia").
• Timeline mode over story clusters; event-propagation explorer (news event → entities → finance exposure).
• LLM explanation layer (optional service, explicitly outside the core).