docs/MASTER_PLAN.md

Vantage — Master Plan

Agentic Market Intelligence. The single source of truth (SSOT) for this project. "What is the market asking for that our research library hasn't covered yet?"

This document is the master reference. Every architectural, scoping, and sequencing decision lives here. Update it as the project evolves; do not let code drift from it.

Table of Contents

1. Product & Problem
2. The Five Capabilities
3. GapScore Methodology
4. Scope
5. Key Trade-offs
6. Tools & Stack
7. Architecture & Patterns
8. Folder Structure
9. Data Model
10. The Agent
11. Edge Cases
12. Build Phases (organic, local-first)
13. Guiding Principles
14. Local Dev Workflow
15. Reproducibility & Delivery
16. Evaluation Methodology
17. Observability
18. Demo Script
19. Effort Summary

---

1. Product & Problem

Research organizations (the case study is framed around a Gartner-like firm) maintain a large internal library of analyst content. The market, meanwhile, moves constantly — new topics surge in search and news every week. The strategic question for research leadership is:

Where is market demand outrunning our coverage?

Vantage answers this by connecting external market demand to internal library coverage through a shared topic vocabulary. For any topic it knows two things at once:

• Momentum — how hot the topic is in the outside world (search/news signal).
• Coverage — how well the internal library addresses it (retrieval density).

The gap between them — the GapScore — drives a ranked commissioning list and a conversational research assistant. Every answer is grounded in cited sources, and every agent decision is fully auditable.

Simulated internal library: we use the public HuffPost News Category Dataset (~200k pre-labeled articles) as a stand-in for the proprietary research corpus. Its built-in category labels give us topic ground truth for free.

---

2. The Five Capabilities

Mapped directly to the case-study brief:

1. Detect emerging topics — from public search/news signals (Google Trends via pytrends).
2. Answer client questions — RAG over the library returning cited sources.
3. Identify gaps — compare topic momentum vs. internal coverage (the core metric).
4. Knowledge Graph — model documents, topics, entities, and their relationships.
5. Agentic orchestration — an agent that plans, retrieves, analyzes, and synthesizes across all of the above.

---

3. GapScore Methodology

The core metric. Borrowed, not invented — adapted from SEO keyword opportunity scoring (Ahrefs/SEMrush content-gap methodology), generalized to research topics.

GapScore(topic) = Momentum_percentile(topic) × (1 − Coverage_percentile(topic))

Why this formula is defensible:

• Both inputs are percentile ranks across all tracked topics → same ordinal scale → multiplication/comparison is dimensionally valid (avoids the trap of subtracting two differently-scaled raw numbers).
• Output is bounded [0, 1], never negative.
• The product (not difference) means both conditions must hold to flag a gap: a topic nobody wants AND that we cover well is correctly scored near 0.

Inputs defined:

• Momentum_percentile — percentile rank of the topic's search-interest z-score (rolling baseline) across all tracked topics. Source: pytrends, cached.
• Coverage_percentile — percentile rank of the topic's retrieval density (count of library chunks above a similarity threshold, normalized) across all tracked topics.

The 2×2 decision matrix (how results are presented to a business audience):

           HIGH COVERAGE
                │
   Over-Served  │  Well-Covered ✅
   (deprioritize)│  (maintain)
────────────────┼──────────────── MOMENTUM →
   Ignore       │  🔴 PRIORITY GAP
   (low demand) │  (commission now)
                │
           LOW COVERAGE

Worked example: topic "Generative AI Enterprise" — momentum_pctile 0.96, coverage_pctile 0.12 → GapScore = 0.96 × (1 − 0.12) = 0.84 → 🔴 Priority Gap.

Union seeding (solves cold start + blind spot): topics are seeded from the union of (a) internal topics (HuffPost categories / extracted keywords) and (b) external trending terms (pytrends top-N). Internal seeding solves the cold-start problem (no external taxonomy needed); external seeding patches the blind spot where we have zero coverage of a surging topic (such topics would never appear if we only looked inward).

---

4. Scope

In scope

• HuffPost dataset as the simulated internal library (~200k pre-labeled articles).
• Hybrid RAG (dense + sparse) → cross-encoder rerank → cited answers.
• Union-seeded gap analysis (internal ∪ external) — cold start + zero-coverage blind spot.
• Knowledge Graph in managed Neo4j; entities via spaCy NER; IPTC-style topic hierarchy.
• LangGraph agent (4 tools) with multi-turn memory + multi-hop reasoning.
• React/TS dashboard; evaluation harness; Langfuse observability; one-command Docker delivery.

Out of scope (with reason)

• BERTopic — HuffPost categories already label topics; nothing to discover.
• GDELT / Guardian / NYT / SEC / SPARQL — size/complexity/auth; documented as production paths.
• Self-hosted DBs — managed free tiers (Qdrant Cloud, Neo4j AuraDB) instead.
• XGBoost/learning-to-rank reranker — needs labeled data; pre-trained cross-encoder wins zero-shot.
• Auth, multi-tenancy, streaming ingestion — V2 items, mentioned in write-up.

---

5. Key Trade-offs

Decision	Chosen	Why	Trade-off accepted
External signal	pytrends (cached)	Free, fast, offline-safe	Lower fidelity than GDELT
Reranker	Pre-trained cross-encoder	No training data needed	Not domain-tuned
Topic seeding	Union (internal ∪ external)	Catches thin and zero coverage	Slightly more compute
Architecture	3-service modular monorepo	CQRS + SSOT, KISS	Not "pure" microservices
Databases	Managed free tiers	Zero ops, zero cost	Free-tier capacity limits
Gap compute	Scheduled/offline, API reads	Never blocks user queries	Gaps eventually-consistent
Topic modeling	HuffPost category labels	Labels already exist	Coarser than learned topics
Delivery	Pre-built GHCR images	One-command boot for reviewer	CI build pipeline upfront cost

---

6. Tools & Stack

Layer	Tool	Role
Frontend	React + Vite + TS + Tailwind + shadcn/ui + react-force-graph	Dashboard UI
API	FastAPI	Read side / BFF + agent host
Agent	LangGraph	Router, 4 tools, ReAct, memory
Domain	vantage_core (shared pkg)	SSOT logic: gapscore, retrieval, topics
Vectors	Qdrant Cloud (free)	Hybrid dense + sparse search (~1M vectors)
Embeddings	all-MiniLM-L6-v2	Chunk + query vectors (384-dim)
Rerank	cross-encoder/ms-marco-MiniLM-L-6-v2	Precision boost (top-50 → top-5)
Graph	Neo4j AuraDB (free)	Topics/entities/gap scores (~200k nodes)
NER	spaCy en_core_web_sm	Entity extraction (ORG/PERSON/GPE)
Signal	pytrends	Momentum (cached, TTL 6h)
LLM	OpenAI gpt-4o-mini	Routing + cited generation
Eval	RAGAS + NDCG@5	Faithfulness, context recall, relevance
Observability	Langfuse Cloud	Trace every call/decision
Delivery	Docker + GHCR + one compose	docker compose up

---

7. Architecture & Patterns

Three services + two managed databases. Local-first during development (plain scripts + start.sh/stop.sh); containerized only at the final phase.

• apps/web — React/TS UI (chat, gap radar, graph). Thin; types generated from OpenAPI.
• apps/api — FastAPI read side (BFF). Hosts the LangGraph agent + 4 tools. Reads only.
• apps/worker — write side. Ingestion, KG build, signal refresh, gap computation.
• packages/vantage_core — SSOT domain: ports & adapters, models, gapscore, retrieval.
• contracts/openapi.yaml — API-first contract → generates the TS client (type SSOT).
• Managed: Qdrant Cloud (vectors), Neo4j AuraDB (graph).

Patterns in play (and what each means here):

• SSOT — Qdrant owns chunks/vectors; Neo4j owns topics/entities/gap scores; signals.json owns external momentum; vantage_core.models owns schemas; openapi.yaml owns API types.
• CQRS-lite — worker writes (ingest, compute gaps); api reads. Gap math never blocks queries.
• Hexagonal (ports & adapters) — domain depends on interfaces; Qdrant/Neo4j/OpenAI/pytrends are swappable adapters (e.g. pytrends → GDELT later).
• Repository — VectorRepo, GraphRepo, SignalSource abstract all I/O; domain is unit-testable.
• Twelve-Factor — config in .env, stateless services, disposable containers, dev/prod parity.
• Cache-aside + TTL — signals.json cached; recompute on miss/expiry.
• Idempotent batch — worker uses upserts; re-running ingestion is safe.
• Graceful degradation — every external dependency has a fallback/health-check.

---

8. Folder Structure

vantage/
├── docs/MASTER_PLAN.md          # this file — the master reference
├── README.md                    # quickstart + architecture diagram (P13)
├── .env.example                 # canonical list of every env var
├── start.sh / stop.sh           # local run (P9–P10): boot/stop api + web
├── Makefile                     # convenience targets (ingest/seed/eval/...)
├── docker-compose.yml           # references GHCR images (P13)
├── .github/workflows/ci.yml     # build+push images to GHCR (P13)
├── contracts/
│   └── openapi.yaml             # API-first contract → TS client (SSOT for types)
├── packages/
│   └── vantage_core/            # SSOT domain logic (imported by api + worker)
│       ├── domain/{gapscore,retrieval,topics}.py
│       ├── adapters/{vector_qdrant,graph_neo4j,signals_pytrends,llm_openai}.py
│       ├── ports.py             # VectorRepo, GraphRepo, SignalSource interfaces
│       ├── models.py            # pydantic domain models (SSOT schemas)
│       └── config.py            # settings from .env
├── apps/
│   ├── api/                     # FastAPI read side + LangGraph agent
│   │   └── vantage_api/{main.py,routers/,agent/{graph,state,tools/}}
│   ├── worker/                  # write side jobs
│   │   └── vantage_worker/{ingest_documents,build_graph,compute_signals,compute_gaps}.py
│   └── web/                     # React + TS dashboard
│       └── src/{features/{chat,gap-radar,graph}/,lib/api/,App.tsx}
├── scripts/                     # one-shot dev scripts (local-first)
│   ├── healthcheck.py  download_data.sh  ingest.py  search.py
│   ├── ask.py  refresh_signals.py  compute_gaps.py  build_graph.py  agent_cli.py
└── eval/{golden_dataset.json,run_eval.py,report.md}

---

9. Data Model

Qdrant (vectors) — one point per chunk:

• id, dense_vector (384-dim), sparse_vector (BM25)
• payload: doc_id, title, category, date, source, chunk_text

Neo4j (graph) — nodes and edges:

• Nodes: (:Document {id,title,category,date}), (:Topic {name,iptc_code,coverage_pctile, momentum_pctile,gap_score,quadrant}), (:Entity {name,type})
• Edges: (Document)-[:MENTIONS]->(Topic), (Document)-[:CONTAINS]->(Entity), (Entity)-[:RELATED_TO]->(Topic), (Topic)-[:BROADER]->(Topic)

signals.json — {fetched_at, topics:[{keyword,z_score,momentum_pctile}]}.

gap_log.jsonl — async query-level log: {query,topic,retrieval_score,momentum_pctile,gap_score,ts}.

---

10. The Agent

LangGraph, ReAct pattern. State carries conversation memory and gap log:

class AgentState(TypedDict):
    messages: Annotated[list, add_messages]   # multi-turn memory
    current_context: str                       # e.g. "APAC", "healthcare"
    last_topics: list[str]                     # topics from previous turn
    gap_log: list[dict]                        # running query-level gap signals

Four tools:

• RAG Tool — Qdrant hybrid → cross-encoder → cited answer.
• Signal Tool — reads signals.json → ranked momentum.
• Gap Tool — Neo4j Cypher → GapScore table + quadrant.
• Graph Tool — Neo4j Cypher → entity/topic/relationship traversal.

Router classifies intent and selects tool(s). Multi-hop chains tools within one query (e.g. Gap → Graph → RAG). Multi-turn memory resolves references like "which of those…". Async side-effect: after each RAG answer with low retrieval score, write to gap_log.jsonl.

---

11. Edge Cases

Representative set (full list maintained alongside tests):

• Zero relevant docs → answer "no internal coverage" + log gap; never hallucinate.
• LLM omits/forges citations → post-validate against retrieved chunk IDs; retry/drop.
• Topic trending, zero coverage → union seeding catches it; coverage_pctile 0 → max gap.
• pytrends 429 / flat series → fall back to cache; guard z-score divide-by-zero.
• Garbage / variant entities → type whitelist + min-frequency + alias normalization.
• AuraDB node limit → cap ingested docs; documented.
• Wrong/missing env var → fail fast with explicit message (health-check).
• Worker hasn't run → API returns "gap data not yet computed" empty-state.
• OpenAI/Langfuse down → backoff / fire-and-forget; never crash the demo.
• Multi-hop latency (10–20s) → FE progress indicator / per-tool status.

---

12. Build Phases (organic, local-first)

Principle: vertical slices — each phase delivers one runnable, testable component. Infra (Docker/GHCR) comes last. Managed DBs are remote endpoints we connect to from local scripts.

Phase 1 — 🔌 Bootstrap & Connectivity (🟢 2h)

• Contains: repo folders, requirements, venv, config.py, .env.example; provision Qdrant Cloud + Neo4j AuraDB; scripts/healthcheck.py.
• Test: python scripts/healthcheck.py → Qdrant OK · Neo4j OK · OpenAI OK.
• Deliverable: green health check. (No Docker, no GHCR.)

Phase 2 — 📚 Data Ingestion (🟡 3–4h)

• Builds on: P1. Contains: download_data.sh, models (Document/Chunk), loader, chunker, embedder, Qdrant adapter (upsert), scripts/ingest.py.
• Test: ingest 2k sample docs → scripts/peek.py shows counts + a sample chunk.
• Deliverable: populated Qdrant collection.

Phase 3 — 🔎 Retrieval (hybrid + rerank) (🟡 3–4h)

• Builds on: P2. Contains: hybrid dense+sparse search, cross-encoder rerank, scripts/search.py (CLI).
• Test: 5 manual queries return on-topic chunks; rerank order sensible.
• Deliverable: retrieval module + search CLI.

Phase 4 — 💬 RAG Answers (cited) (🟡 3–4h)

• Builds on: P3. Contains: prompt template, OpenAI adapter, citation formatter + validation, scripts/ask.py.
• Test: ask 5 questions → real citations; "no coverage" path triggers correctly.
• Deliverable: working RAG via CLI.

Phase 5 — 📈 External Signals (🟢 2–3h)

• Builds on: P1. Contains: pytrends adapter, z-score→percentile, cache-aside TTL → signals.json, scripts/refresh_signals.py.
• Test: run → signals.json populated; rate-limit falls back to cache.
• Deliverable: momentum data.

Phase 6 — 🎯 Coverage + GapScore (🟡 3–4h)

• Builds on: P2, P5. Contains: coverage via Qdrant density, union seeding + normalization, gapscore.py, scripts/compute_gaps.py.
• Test: gap table prints; zero-coverage topic → max gap; unit tests on gapscore.
• Deliverable: gap_scores.csv.

Phase 7 — 🕸️ Knowledge Graph (🟡 3–4h)

• Builds on: P2, P6. Contains: spaCy NER + entity filtering/normalization, Neo4j adapter (Doc/Topic/Entity upserts + edges), write gap scores onto Topic nodes, scripts/build_graph.py.
• Test: Cypher in Neo4j Browser returns top gap topics + linked entities.
• Deliverable: populated graph.

Phase 8 — 🤖 Agent (CLI) (🔴 4–5h)

• Builds on: P4, P5, P6, P7. Contains: LangGraph state (memory), router, 4 tools, ReAct multi-hop, scripts/agent_cli.py.
• Test: all 5 demo queries → correct routing + multi-hop + memory.
• Deliverable: working agent, terminal-only.

Phase 9 — 🌐 API (🟡 3–4h)

• Builds on: P8. Contains: FastAPI routers (chat/gaps/graph), session-keyed memory, emit openapi.yaml, start.sh/stop.sh (local uvicorn).
• Test: curl each endpoint; openapi.yaml generated.
• Deliverable: API running locally via start.sh.

Phase 10 — 🎨 Frontend (🔴 6–8h)

• Builds on: P9. Contains: React/Vite/TS + Tailwind + shadcn, 3 features (chat/gap-radar/graph), generated TS client; start.sh boots api + web, stop.sh stops both.
• Test: click through 3 tabs; all 5 queries work in the UI.
• Deliverable: full local app via start.sh.

Phase 11 — 📊 Evaluation (🟢 2–3h)

• Builds on: P4. Contains: golden set (10 hand + 40 Opus), RAGAS + NDCG@5 → eval/report.md.
• Test: python eval/run_eval.py prints metrics.
• Deliverable: metrics report for README.

Phase 12 — 🔭 Observability (🟢 2h)

• Builds on: P8. Contains: Langfuse wiring across LLM/tool/agent calls (fire-and-forget).
• Test: traces appear; app works if Langfuse is down.
• Deliverable: traced runs.

Phase 13 — 📦 Containerize + GHCR + One-Command (🟡 3–4h, the very end)

• Builds on: all. Contains: 3 Dockerfiles, docker-compose.yml (GHCR images), healthchecks + depends_on, GitHub Action build/push, README + architecture diagram.
• Test: clean machine → docker compose up → everything boots.
• Deliverable: submission-ready reproducible repo.

---

13. Guiding Principles

Principle	How it's honored
One thing at a time	Each phase = a single capability, nothing bundled
Always runnable	Every phase ends with a script/CLI you can execute
Local-first	Plain scripts; start.sh/stop.sh at P9–P10; Docker/GHCR last (P13)
Builds upward	Search → RAG → Signals → Gap → Graph → Agent → API → UI
Testable atoms	Each phase has a concrete pass/fail test
De-risked	Hardest integration (agent, P8) sits on already-proven tools
SSOT everywhere	Domain in vantage_core; types in openapi.yaml; no duplicated logic

---

14. Local Dev Workflow

• Dependencies: Python 3.11 venv per service; pip install -e packages/vantage_core.
• Config: copy .env.example → .env; fill Qdrant/Neo4j/OpenAI/Langfuse creds.
• One-shot scripts (P2–P8): ingest.py, search.py, ask.py, refresh_signals.py, compute_gaps.py, build_graph.py, agent_cli.py — run individual capabilities.
• Long-running services (P9–P10): ./start.sh boots api (uvicorn) + web (vite); ./stop.sh stops both. No Docker required during development.
• Seeding: make seed (or scripts/seed_all.sh) populates the managed DBs end-to-end, idempotently — runnable by reviewer if they bring their own free-tier creds.

---

15. Reproducibility & Delivery

• Images: built and pushed to GHCR (ghcr.io/<user>/vantage-{api,worker,web}) by a GitHub Action on merge to main.
• One command: reviewer clones, drops in provided .env (demo creds), runs docker compose up → Docker pulls pre-built images → full stack boots. DBs are pre-seeded.
• Rule: always read docker-compose.yml and .env.example before answering infra questions; never guess env var names or ports.

---

16. Evaluation Methodology

• Golden set: 50 Q&A — 10 hand-labeled (human ground truth, breaks LLM-circularity)
  • 40 generated by Claude Opus over sampled articles.
• Metrics: RAGAS (faithfulness, context recall, answer relevance) + NDCG@5 for retrieval.
• Output: eval/report.md table, surfaced in the README.
• Honesty: LLM-as-judge limitations stated explicitly; the 10 human-labeled anchor credibility.

---

17. Observability

• Langfuse Cloud traces every LLM call, tool call, and agent handoff.
• Logged per query: query → extracted topic → momentum → retrieval score → answer → gap flag.
• Fire-and-forget: tracing never blocks or breaks a user response.

---

18. Demo Script

Five rehearsed queries, in order — each builds on the last and together they exercise every capability in the brief:

1. RAG only — "What are the challenges of AI in healthcare?" → cited answer.
2. Signal only — "What technology topics are surging right now?" → ranked momentum.
3. Gap only — "What should we commission next quarter?" → GapScore table + 2×2.
4. Multi-hop (Gap→Graph→RAG) — "Which companies sit in our biggest AI gaps?" → synthesis.
5. Multi-turn memory — follow-up "Which of those are we thin on?" → memory-scoped gaps.

---

19. Effort Summary

Phase	Theme	Effort
1	Bootstrap & Connectivity	🟢 2h
2	Data Ingestion	🟡 3–4h
3	Retrieval (hybrid + rerank)	🟡 3–4h
4	RAG Answers (cited)	🟡 3–4h
5	External Signals	🟢 2–3h
6	Coverage + GapScore	🟡 3–4h
7	Knowledge Graph	🟡 3–4h
8	Agent (CLI)	🔴 4–5h
9	API	🟡 3–4h
10	Frontend	🔴 6–8h
11	Evaluation	🟢 2–3h
12	Observability	🟢 2h
13	Containerize + GHCR	🟡 3–4h
Total		≈ 42–53h

Build order is strictly sequential P1 → P13. Fits comfortably within the 2–3 day budget.