docs/MASTER_PLAN.md

# Vantage — Master Plan

> **Agentic Market Intelligence.** The single source of truth (SSOT) for this project.
> *"What is the market asking for that our research library hasn't covered yet?"*
>
> This document is the master reference. Every architectural, scoping, and sequencing
> decision lives here. Update it as the project evolves; do not let code drift from it.

## Table of Contents
1. [Product & Problem](#1-product--problem)
2. [The Five Capabilities](#2-the-five-capabilities)
3. [GapScore Methodology](#3-gapscore-methodology)
4. [Scope](#4-scope)
5. [Key Trade-offs](#5-key-trade-offs)
6. [Tools & Stack](#6-tools--stack)
7. [Architecture & Patterns](#7-architecture--patterns)
8. [Folder Structure](#8-folder-structure)
9. [Data Model](#9-data-model)
10. [The Agent](#10-the-agent)
11. [Edge Cases](#11-edge-cases)
12. [Build Phases (organic, local-first)](#12-build-phases-organic-local-first)
13. [Guiding Principles](#13-guiding-principles)
14. [Local Dev Workflow](#14-local-dev-workflow)
15. [Reproducibility & Delivery](#15-reproducibility--delivery)
16. [Evaluation Methodology](#16-evaluation-methodology)
17. [Observability](#17-observability)
18. [Demo Script](#18-demo-script)
19. [Effort Summary](#19-effort-summary)

---

## 1. Product & Problem

Research organizations (the case study is framed around a Gartner-like firm) maintain a large
internal library of analyst content. The market, meanwhile, moves constantly — new topics surge
in search and news every week. The strategic question for research leadership is:

> **Where is market demand outrunning our coverage?**

**Vantage** answers this by connecting **external market demand** to **internal library
coverage** through a shared topic vocabulary. For any topic it knows two things at once:
- **Momentum** — how hot the topic is in the outside world (search/news signal).
- **Coverage** — how well the internal library addresses it (retrieval density).

The gap between them — the **GapScore** — drives a ranked commissioning list and a
conversational research assistant. Every answer is grounded in cited sources, and every agent
decision is fully auditable.

**Simulated internal library:** we use the public **HuffPost News Category Dataset**
(~200k pre-labeled articles) as a stand-in for the proprietary research corpus. Its built-in
category labels give us topic ground truth for free.

---

## 2. The Five Capabilities

Mapped directly to the case-study brief:

1. **Detect emerging topics** — from public search/news signals (Google Trends via pytrends).
2. **Answer client questions** — RAG over the library returning **cited** sources.
3. **Identify gaps** — compare topic momentum vs. internal coverage (the core metric).
4. **Knowledge Graph** — model documents, topics, entities, and their relationships.
5. **Agentic orchestration** — an agent that plans, retrieves, analyzes, and synthesizes
   across all of the above.

---

## 3. GapScore Methodology

The core metric. **Borrowed, not invented** — adapted from SEO keyword opportunity scoring
(Ahrefs/SEMrush content-gap methodology), generalized to research topics.

```
GapScore(topic) = Momentum_percentile(topic) × (1 − Coverage_percentile(topic))
```

**Why this formula is defensible:**
- Both inputs are **percentile ranks** across all tracked topics → same ordinal scale →
  multiplication/comparison is dimensionally valid (avoids the trap of subtracting two
  differently-scaled raw numbers).
- Output is bounded **[0, 1]**, never negative.
- The **product** (not difference) means *both* conditions must hold to flag a gap: a topic
  nobody wants AND that we cover well is correctly scored near 0.

**Inputs defined:**
- **Momentum_percentile** — percentile rank of the topic's search-interest z-score
  (rolling baseline) across all tracked topics. Source: pytrends, cached.
- **Coverage_percentile** — percentile rank of the topic's *retrieval density* (count of
  library chunks above a similarity threshold, normalized) across all tracked topics.

**The 2×2 decision matrix** (how results are presented to a business audience):

```
           HIGH COVERAGE
                │
   Over-Served  │  Well-Covered ✅
   (deprioritize)│  (maintain)
────────────────┼──────────────── MOMENTUM →
   Ignore       │  🔴 PRIORITY GAP
   (low demand) │  (commission now)
                │
           LOW COVERAGE
```

**Worked example:** topic "Generative AI Enterprise" — momentum_pctile 0.96, coverage_pctile
0.12 → GapScore = 0.96 × (1 − 0.12) = **0.84** → 🔴 Priority Gap.

**Union seeding (solves cold start + blind spot):** topics are seeded from the **union** of
(a) internal topics (HuffPost categories / extracted keywords) and (b) external trending terms
(pytrends top-N). Internal seeding solves the cold-start problem (no external taxonomy needed);
external seeding patches the blind spot where we have **zero** coverage of a surging topic (such
topics would never appear if we only looked inward).

---

## 4. Scope

**In scope**
- HuffPost dataset as the *simulated internal library* (~200k pre-labeled articles).
- Hybrid RAG (dense + sparse) → cross-encoder rerank → cited answers.
- **Union-seeded** gap analysis (internal ∪ external) — cold start + zero-coverage blind spot.
- Knowledge Graph in managed Neo4j; entities via spaCy NER; IPTC-style topic hierarchy.
- LangGraph agent (4 tools) with multi-turn memory + multi-hop reasoning.
- React/TS dashboard; evaluation harness; Langfuse observability; one-command Docker delivery.

**Out of scope (with reason)**
- **BERTopic** — HuffPost categories already label topics; nothing to discover.
- **GDELT / Guardian / NYT / SEC / SPARQL** — size/complexity/auth; documented as production paths.
- **Self-hosted DBs** — managed free tiers (Qdrant Cloud, Neo4j AuraDB) instead.
- **XGBoost/learning-to-rank reranker** — needs labeled data; pre-trained cross-encoder wins zero-shot.
- **Auth, multi-tenancy, streaming ingestion** — V2 items, mentioned in write-up.

---

## 5. Key Trade-offs

| Decision | Chosen | Why | Trade-off accepted |
|---|---|---|---|
| External signal | pytrends (cached) | Free, fast, offline-safe | Lower fidelity than GDELT |
| Reranker | Pre-trained cross-encoder | No training data needed | Not domain-tuned |
| Topic seeding | Union (internal ∪ external) | Catches thin *and* zero coverage | Slightly more compute |
| Architecture | 3-service modular monorepo | CQRS + SSOT, KISS | Not "pure" microservices |
| Databases | Managed free tiers | Zero ops, zero cost | Free-tier capacity limits |
| Gap compute | Scheduled/offline, API reads | Never blocks user queries | Gaps eventually-consistent |
| Topic modeling | HuffPost category labels | Labels already exist | Coarser than learned topics |
| Delivery | Pre-built GHCR images | One-command boot for reviewer | CI build pipeline upfront cost |

---

## 6. Tools & Stack

| Layer | Tool | Role |
|---|---|---|
| Frontend | React + Vite + TS + Tailwind + shadcn/ui + react-force-graph | Dashboard UI |
| API | FastAPI | Read side / BFF + agent host |
| Agent | LangGraph | Router, 4 tools, ReAct, memory |
| Domain | `vantage_core` (shared pkg) | SSOT logic: gapscore, retrieval, topics |
| Vectors | **Qdrant Cloud (free)** | Hybrid dense + sparse search (~1M vectors) |
| Embeddings | all-MiniLM-L6-v2 | Chunk + query vectors (384-dim) |
| Rerank | cross-encoder/ms-marco-MiniLM-L-6-v2 | Precision boost (top-50 → top-5) |
| Graph | **Neo4j AuraDB (free)** | Topics/entities/gap scores (~200k nodes) |
| NER | spaCy `en_core_web_sm` | Entity extraction (ORG/PERSON/GPE) |
| Signal | pytrends | Momentum (cached, TTL 6h) |
| LLM | OpenAI `gpt-4o-mini` | Routing + cited generation |
| Eval | RAGAS + NDCG@5 | Faithfulness, context recall, relevance |
| Observability | Langfuse Cloud | Trace every call/decision |
| Delivery | Docker + GHCR + one compose | `docker compose up` |

---

## 7. Architecture & Patterns

**Three services + two managed databases.** Local-first during development (plain scripts +
`start.sh`/`stop.sh`); containerized only at the final phase.

- **`apps/web`** — React/TS UI (chat, gap radar, graph). Thin; types generated from OpenAPI.
- **`apps/api`** — FastAPI **read side** (BFF). Hosts the LangGraph agent + 4 tools. Reads only.
- **`apps/worker`** — **write side**. Ingestion, KG build, signal refresh, gap computation.
- **`packages/vantage_core`** — **SSOT domain**: ports & adapters, models, gapscore, retrieval.
- **`contracts/openapi.yaml`** — API-first contract → generates the TS client (type SSOT).
- **Managed:** Qdrant Cloud (vectors), Neo4j AuraDB (graph).

**Patterns in play (and what each means here):**
- **SSOT** — Qdrant owns chunks/vectors; Neo4j owns topics/entities/gap scores; `signals.json`
  owns external momentum; `vantage_core.models` owns schemas; `openapi.yaml` owns API types.
- **CQRS-lite** — `worker` writes (ingest, compute gaps); `api` reads. Gap math never blocks queries.
- **Hexagonal (ports & adapters)** — domain depends on interfaces; Qdrant/Neo4j/OpenAI/pytrends
  are swappable adapters (e.g. pytrends → GDELT later).
- **Repository** — `VectorRepo`, `GraphRepo`, `SignalSource` abstract all I/O; domain is unit-testable.
- **Twelve-Factor** — config in `.env`, stateless services, disposable containers, dev/prod parity.
- **Cache-aside + TTL** — `signals.json` cached; recompute on miss/expiry.
- **Idempotent batch** — worker uses upserts; re-running ingestion is safe.
- **Graceful degradation** — every external dependency has a fallback/health-check.

---

## 8. Folder Structure

```
vantage/
├── docs/MASTER_PLAN.md          # this file — the master reference
├── README.md                    # quickstart + architecture diagram (P13)
├── .env.example                 # canonical list of every env var
├── start.sh / stop.sh           # local run (P9–P10): boot/stop api + web
├── Makefile                     # convenience targets (ingest/seed/eval/...)
├── docker-compose.yml           # references GHCR images (P13)
├── .github/workflows/ci.yml     # build+push images to GHCR (P13)
├── contracts/
│   └── openapi.yaml             # API-first contract → TS client (SSOT for types)
├── packages/
│   └── vantage_core/            # SSOT domain logic (imported by api + worker)
│       ├── domain/{gapscore,retrieval,topics}.py
│       ├── adapters/{vector_qdrant,graph_neo4j,signals_pytrends,llm_openai}.py
│       ├── ports.py             # VectorRepo, GraphRepo, SignalSource interfaces
│       ├── models.py            # pydantic domain models (SSOT schemas)
│       └── config.py            # settings from .env
├── apps/
│   ├── api/                     # FastAPI read side + LangGraph agent
│   │   └── vantage_api/{main.py,routers/,agent/{graph,state,tools/}}
│   ├── worker/                  # write side jobs
│   │   └── vantage_worker/{ingest_documents,build_graph,compute_signals,compute_gaps}.py
│   └── web/                     # React + TS dashboard
│       └── src/{features/{chat,gap-radar,graph}/,lib/api/,App.tsx}
├── scripts/                     # one-shot dev scripts (local-first)
│   ├── healthcheck.py  download_data.sh  ingest.py  search.py
│   ├── ask.py  refresh_signals.py  compute_gaps.py  build_graph.py  agent_cli.py
└── eval/{golden_dataset.json,run_eval.py,report.md}
```

---

## 9. Data Model

**Qdrant (vectors)** — one point per chunk:
- `id`, `dense_vector` (384-dim), `sparse_vector` (BM25)
- payload: `doc_id`, `title`, `category`, `date`, `source`, `chunk_text`

**Neo4j (graph)** — nodes and edges:
- Nodes: `(:Document {id,title,category,date})`, `(:Topic {name,iptc_code,coverage_pctile,
  momentum_pctile,gap_score,quadrant})`, `(:Entity {name,type})`
- Edges: `(Document)-[:MENTIONS]->(Topic)`, `(Document)-[:CONTAINS]->(Entity)`,
  `(Entity)-[:RELATED_TO]->(Topic)`, `(Topic)-[:BROADER]->(Topic)`

**`signals.json`** — `{fetched_at, topics:[{keyword,z_score,momentum_pctile}]}`.

**`gap_log.jsonl`** — async query-level log: `{query,topic,retrieval_score,momentum_pctile,gap_score,ts}`.

---

## 10. The Agent

**LangGraph, ReAct pattern.** State carries conversation memory and gap log:
```python
class AgentState(TypedDict):
    messages: Annotated[list, add_messages]   # multi-turn memory
    current_context: str                       # e.g. "APAC", "healthcare"
    last_topics: list[str]                     # topics from previous turn
    gap_log: list[dict]                        # running query-level gap signals
```

**Four tools:**
- **RAG Tool** — Qdrant hybrid → cross-encoder → cited answer.
- **Signal Tool** — reads `signals.json` → ranked momentum.
- **Gap Tool** — Neo4j Cypher → GapScore table + quadrant.
- **Graph Tool** — Neo4j Cypher → entity/topic/relationship traversal.

**Router** classifies intent and selects tool(s). **Multi-hop** chains tools within one query
(e.g. Gap → Graph → RAG). **Multi-turn memory** resolves references like "which of those…".
**Async side-effect:** after each RAG answer with low retrieval score, write to `gap_log.jsonl`.

---

## 11. Edge Cases

Representative set (full list maintained alongside tests):
- **Zero relevant docs** → answer "no internal coverage" + log gap; never hallucinate.
- **LLM omits/forges citations** → post-validate against retrieved chunk IDs; retry/drop.
- **Topic trending, zero coverage** → union seeding catches it; coverage_pctile 0 → max gap.
- **pytrends 429 / flat series** → fall back to cache; guard z-score divide-by-zero.
- **Garbage / variant entities** → type whitelist + min-frequency + alias normalization.
- **AuraDB node limit** → cap ingested docs; documented.
- **Wrong/missing env var** → fail fast with explicit message (health-check).
- **Worker hasn't run** → API returns "gap data not yet computed" empty-state.
- **OpenAI/Langfuse down** → backoff / fire-and-forget; never crash the demo.
- **Multi-hop latency (10–20s)** → FE progress indicator / per-tool status.

---

## 12. Build Phases (organic, local-first)

**Principle:** vertical slices — each phase delivers *one* runnable, testable component. Infra
(Docker/GHCR) comes last. Managed DBs are remote endpoints we connect to from local scripts.

### Phase 1 — 🔌 Bootstrap & Connectivity *(🟢 2h)*
- **Contains:** repo folders, `requirements`, venv, `config.py`, `.env.example`; provision
  Qdrant Cloud + Neo4j AuraDB; `scripts/healthcheck.py`.
- **Test:** `python scripts/healthcheck.py` → `Qdrant OK · Neo4j OK · OpenAI OK`.
- **Deliverable:** green health check. *(No Docker, no GHCR.)*

### Phase 2 — 📚 Data Ingestion *(🟡 3–4h)*
- **Builds on:** P1. **Contains:** `download_data.sh`, `models` (Document/Chunk), loader,
  chunker, embedder, Qdrant adapter (upsert), `scripts/ingest.py`.
- **Test:** ingest 2k sample docs → `scripts/peek.py` shows counts + a sample chunk.
- **Deliverable:** populated Qdrant collection.

### Phase 3 — 🔎 Retrieval (hybrid + rerank) *(🟡 3–4h)*
- **Builds on:** P2. **Contains:** hybrid dense+sparse search, cross-encoder rerank,
  `scripts/search.py` (CLI).
- **Test:** 5 manual queries return on-topic chunks; rerank order sensible.
- **Deliverable:** `retrieval` module + search CLI.

### Phase 4 — 💬 RAG Answers (cited) *(🟡 3–4h)*
- **Builds on:** P3. **Contains:** prompt template, OpenAI adapter, citation formatter +
  validation, `scripts/ask.py`.
- **Test:** ask 5 questions → real citations; "no coverage" path triggers correctly.
- **Deliverable:** working RAG via CLI.

### Phase 5 — 📈 External Signals *(🟢 2–3h)*
- **Builds on:** P1. **Contains:** pytrends adapter, z-score→percentile, cache-aside TTL →
  `signals.json`, `scripts/refresh_signals.py`.
- **Test:** run → `signals.json` populated; rate-limit falls back to cache.
- **Deliverable:** momentum data.

### Phase 6 — 🎯 Coverage + GapScore *(🟡 3–4h)*
- **Builds on:** P2, P5. **Contains:** coverage via Qdrant density, **union seeding** +
  normalization, `gapscore.py`, `scripts/compute_gaps.py`.
- **Test:** gap table prints; zero-coverage topic → max gap; **unit tests** on `gapscore`.
- **Deliverable:** `gap_scores.csv`.

### Phase 7 — 🕸️ Knowledge Graph *(🟡 3–4h)*
- **Builds on:** P2, P6. **Contains:** spaCy NER + entity filtering/normalization, Neo4j
  adapter (Doc/Topic/Entity upserts + edges), write gap scores onto Topic nodes,
  `scripts/build_graph.py`.
- **Test:** Cypher in Neo4j Browser returns top gap topics + linked entities.
- **Deliverable:** populated graph.

### Phase 8 — 🤖 Agent (CLI) *(🔴 4–5h)*
- **Builds on:** P4, P5, P6, P7. **Contains:** LangGraph state (memory), router, 4 tools,
  ReAct multi-hop, `scripts/agent_cli.py`.
- **Test:** all 5 demo queries → correct routing + multi-hop + memory.
- **Deliverable:** working agent, terminal-only.

### Phase 9 — 🌐 API *(🟡 3–4h)*
- **Builds on:** P8. **Contains:** FastAPI routers (chat/gaps/graph), session-keyed memory,
  emit `openapi.yaml`, `start.sh`/`stop.sh` (local uvicorn).
- **Test:** `curl` each endpoint; `openapi.yaml` generated.
- **Deliverable:** API running locally via `start.sh`.

### Phase 10 — 🎨 Frontend *(🔴 6–8h)*
- **Builds on:** P9. **Contains:** React/Vite/TS + Tailwind + shadcn, 3 features
  (chat/gap-radar/graph), generated TS client; `start.sh` boots **api + web**, `stop.sh` stops both.
- **Test:** click through 3 tabs; all 5 queries work in the UI.
- **Deliverable:** full local app via `start.sh`.

### Phase 11 — 📊 Evaluation *(🟢 2–3h)*
- **Builds on:** P4. **Contains:** golden set (10 hand + 40 Opus), RAGAS + NDCG@5 → `eval/report.md`.
- **Test:** `python eval/run_eval.py` prints metrics.
- **Deliverable:** metrics report for README.

### Phase 12 — 🔭 Observability *(🟢 2h)*
- **Builds on:** P8. **Contains:** Langfuse wiring across LLM/tool/agent calls (fire-and-forget).
- **Test:** traces appear; app works if Langfuse is down.
- **Deliverable:** traced runs.

### Phase 13 — 📦 Containerize + GHCR + One-Command *(🟡 3–4h, the very end)*
- **Builds on:** all. **Contains:** 3 Dockerfiles, `docker-compose.yml` (GHCR images),
  healthchecks + `depends_on`, GitHub Action build/push, README + architecture diagram.
- **Test:** clean machine → `docker compose up` → everything boots.
- **Deliverable:** submission-ready reproducible repo.

---

## 13. Guiding Principles

| Principle | How it's honored |
|---|---|
| One thing at a time | Each phase = a single capability, nothing bundled |
| Always runnable | Every phase ends with a script/CLI you can execute |
| Local-first | Plain scripts; `start.sh`/`stop.sh` at P9–P10; Docker/GHCR last (P13) |
| Builds upward | Search → RAG → Signals → Gap → Graph → Agent → API → UI |
| Testable atoms | Each phase has a concrete pass/fail test |
| De-risked | Hardest integration (agent, P8) sits on already-proven tools |
| SSOT everywhere | Domain in `vantage_core`; types in `openapi.yaml`; no duplicated logic |

---

## 14. Local Dev Workflow

- **Dependencies:** Python 3.11 venv per service; `pip install -e packages/vantage_core`.
- **Config:** copy `.env.example` → `.env`; fill Qdrant/Neo4j/OpenAI/Langfuse creds.
- **One-shot scripts (P2–P8):** `ingest.py`, `search.py`, `ask.py`, `refresh_signals.py`,
  `compute_gaps.py`, `build_graph.py`, `agent_cli.py` — run individual capabilities.
- **Long-running services (P9–P10):** `./start.sh` boots api (uvicorn) + web (vite);
  `./stop.sh` stops both. No Docker required during development.
- **Seeding:** `make seed` (or `scripts/seed_all.sh`) populates the managed DBs end-to-end,
  idempotently — runnable by reviewer if they bring their own free-tier creds.

---

## 15. Reproducibility & Delivery

- **Images:** built and pushed to **GHCR** (`ghcr.io/<user>/vantage-{api,worker,web}`) by a
  GitHub Action on merge to main.
- **One command:** reviewer clones, drops in provided `.env` (demo creds), runs
  `docker compose up` → Docker pulls pre-built images → full stack boots. DBs are pre-seeded.
- **Rule:** always read `docker-compose.yml` and `.env.example` before answering infra
  questions; never guess env var names or ports.

---

## 16. Evaluation Methodology

- **Golden set:** 50 Q&A — **10 hand-labeled** (human ground truth, breaks LLM-circularity)
  + 40 generated by Claude Opus over sampled articles.
- **Metrics:** RAGAS (faithfulness, context recall, answer relevance) + **NDCG@5** for retrieval.
- **Output:** `eval/report.md` table, surfaced in the README.
- **Honesty:** LLM-as-judge limitations stated explicitly; the 10 human-labeled anchor credibility.

---

## 17. Observability

- **Langfuse Cloud** traces every LLM call, tool call, and agent handoff.
- Logged per query: query → extracted topic → momentum → retrieval score → answer → gap flag.
- **Fire-and-forget:** tracing never blocks or breaks a user response.

---

## 18. Demo Script

Five rehearsed queries, in order — each builds on the last and together they exercise every
capability in the brief:
1. **RAG only** — *"What are the challenges of AI in healthcare?"* → cited answer.
2. **Signal only** — *"What technology topics are surging right now?"* → ranked momentum.
3. **Gap only** — *"What should we commission next quarter?"* → GapScore table + 2×2.
4. **Multi-hop** (Gap→Graph→RAG) — *"Which companies sit in our biggest AI gaps?"* → synthesis.
5. **Multi-turn memory** — follow-up *"Which of those are we thin on?"* → memory-scoped gaps.

---

## 19. Effort Summary

| Phase | Theme | Effort |
|---|---|---|
| 1 | Bootstrap & Connectivity | 🟢 2h |
| 2 | Data Ingestion | 🟡 3–4h |
| 3 | Retrieval (hybrid + rerank) | 🟡 3–4h |
| 4 | RAG Answers (cited) | 🟡 3–4h |
| 5 | External Signals | 🟢 2–3h |
| 6 | Coverage + GapScore | 🟡 3–4h |
| 7 | Knowledge Graph | 🟡 3–4h |
| 8 | Agent (CLI) | 🔴 4–5h |
| 9 | API | 🟡 3–4h |
| 10 | Frontend | 🔴 6–8h |
| 11 | Evaluation | 🟢 2–3h |
| 12 | Observability | 🟢 2h |
| 13 | Containerize + GHCR | 🟡 3–4h |
| **Total** | | **≈ 42–53h** |

Build order is strictly sequential P1 → P13. Fits comfortably within the 2–3 day budget.