README.md

metadata

title: Vantage
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false

Vantage — Agentic Market Intelligence

Vantage is an agentic RAG system that surfaces emerging content gaps — topics the market is demanding but your corpus barely covers. It fuses live demand signals with vector coverage and a knowledge graph, then lets a LangGraph agent reason over all three.

• Google Trends signals — demand volume + velocity per topic
• Qdrant — hybrid (dense + sparse) vector search over a ~200k-doc corpus
• Neo4j — knowledge graph of (:Topic)-[:COVERED_BY]->(:Document) edges
• LangGraph agent — orchestrates gap analysis, RAG search, and signals with multi-turn memory

Gap score = Demand × (1 − Coverage) — high demand, low coverage ⇒ a gap worth filling.

---

Architecture

Vantage runs as two planes that share the same core library and the same two databases: an offline pipeline that populates Qdrant + Neo4j, and an online service that only reads them. The app never writes corpus data at request time — it serves what the pipeline already built.

Serving plane (what runs at request time)

flowchart LR
    User([Browser]) --> Web[React + Vite UI]
    Web -->|REST /api| API[FastAPI<br/>chat · gaps · graph · signals]
    API --> Agent[LangGraph ReAct agent]

    Agent -->|rag_search| RAG[Hybrid retrieval<br/>+ rerank]
    Agent -->|gap_analysis · trending_topics<br/>coverage_lookup · graph_lookup| GRAPH[(Neo4j<br/>Topic→Document)]
    Agent -->|estimate_gap| LIVE[Live Trends + coverage]

    RAG --> QDRANT[(Qdrant<br/>dense + sparse)]
    RAG --> OPENAI{{OpenAI<br/>gpt-4o-mini}}
    LIVE --> TRENDS{{Google Trends}}
    LIVE --> QDRANT

Request lifecycle: the UI calls a FastAPI route → the agent picks tools from their docstrings → each tool runs through a domain port (rag, retrieval, gap, coverage, topic_index) backed by an adapter (Qdrant, Neo4j, OpenAI, sentence-transformers, pytrends) → the agent synthesises a grounded answer, citing sources and surfacing the numbers.

Ingestion plane (offline — run once, before serving)

flowchart LR
    SEEDS[build_gartner_seeds<br/>+ mine_gdelt_topics] --> TOPICS[(topic vocab)]
    CORPUS[download_data] --> INGEST[ingest.py<br/>chunk + embed]
    INGEST --> QDRANT[(Qdrant)]
    TOPICS --> TAG[tag_topics] --> COV[topic_coverage.csv]
    TOPICS --> TIDX[ingest_topics] --> QDRANT
    COV --> BUILD[build_graph<br/>coverage × signals × gap] --> NEO[(Neo4j)]

This plane is bash/Unix-only and is what Windows users cannot run (see the Windows note at the end).

Layering

All business logic lives in packages/vantage_core and follows a hexagonal design:

Layer	Location	Responsibility
Ports (domain)	vantage_core/domain/	What the app does — retrieval, RAG, gap & coverage scoring — independent of any vendor
Adapters	vantage_core/adapters/	How it's done — Qdrant, Neo4j, OpenAI, sentence-transformers, pytrends
Agent	vantage_core/agent/	LangGraph ReAct loop + the six tools that wrap the ports
Entry points	apps/api, scripts/	Thin shells: the HTTP API and the pipeline/dev CLIs

---

Prerequisites

You need accounts / endpoints for four external services (all have free tiers):

Service	Used for	Required
Qdrant Cloud	Vector search	✅
Neo4j AuraDB	Knowledge graph	✅
OpenAI	LLM (default gpt-4o-mini)	✅
Langfuse Cloud	LLM tracing	optional

Plus: Python 3.11, Node 20+, and (optionally) Docker.

---

Setup

# 1. Create the Python env (conda) and install the package + API deps
conda env create -f environment.yaml      # creates the 'py311' env
conda activate py311
make install                              # pip install -e vantage_core + API reqs

# 2. Configure credentials
cp .env.example .env
$EDITOR .env                              # fill in Qdrant / Neo4j / OpenAI keys

# 3. Verify every external service is reachable (pass/fail gate)
make healthcheck

make healthcheck pings Qdrant, Neo4j, OpenAI, and Langfuse and prints a colour-coded status line for each. Don't proceed until it's green.

---

Data pipeline — run this once, in order

This populates Qdrant and Neo4j. Every step is idempotent (safe to re-run). The app won't return anything until these have run, because start.sh only serves what's already in the databases.

# ── 1. TOPICS — build the topic vocabulary (the "get topics" step) ────────────
python scripts/build_gartner_seeds.py     # → seeds/topics.csv + data/topics_seed.json
python scripts/mine_gdelt_topics.py       # augment with fresh GDELT demand-side topics (BigQuery)

# ── 2. DATA — fetch the HuffPost corpus into ./data (gitignored) ──────────────
bash scripts/download_data.sh             # see script header for auth options

# ── 3. INGEST — chunk + embed docs → Qdrant ──────────────────────────────────
python scripts/ingest.py                  # honours INGEST_LIMIT (default 2000; 0 = full)
python scripts/peek.py                    # verify: point count + sample payload

# ── 4. TAG — label each chunk with the topics it mentions ─────────────────────
python scripts/tag_topics.py              # → topic_coverage.csv (read by step 6)

# ── 5. TOPIC INDEX — embed the vocabulary into its own collection ─────────────
python scripts/ingest_topics.py           # → vantage_topics collection (semantic gap match)

# ── 6. GRAPH — coverage × signals × gap score → Neo4j ─────────────────────────
python scripts/build_graph.py             # builds (:Topic)-[:COVERED_BY]->(:Document)

Order matters at two points: topics (1) must exist before tagging (4) and the topic index (5); the graph (6) reads topic_coverage.csv from tagging (4). Everything else is independent. To start the graph over, run python scripts/wipe_graph.py first.

Quick sanity checks (read-only, no writes):

python scripts/gap_report.py              # ranked GapScore table
python scripts/search.py "your query"     # hybrid retrieval + rerank
python scripts/ask.py "your question"     # full RAG answer with citations
python scripts/agent_cli.py               # interactive agent REPL
python scripts/demo_queries.py            # 4 canned end-to-end agent queries

---

Run the app

Once the pipeline above has populated the databases, start the services. This is the last step.

make start          # or: bash start.sh

• API → http://localhost:8000 (FastAPI + agent; logs in api.log)
• Web → http://localhost:5173 (React/Vite UI; logs in web.log)

start.sh stops anything stale on ports 8000/5173, launches the API, polls its healthcheck until ready, then launches the frontend. To stop everything:

make stop           # or: bash stop.sh

Note: start.sh currently invokes Python via a hard-coded conda path (/Users/sourabh/opt/miniconda3/envs/py311/bin/python). If your env lives elsewhere, edit that line or run uvicorn directly: python -m uvicorn apps.api.main:app --port 8000 --reload.

---

Run with Docker (single container)

The Dockerfile builds the frontend, bakes the ML models in for offline inference, and serves the API + static UI from one image on port 7860 (Hugging Face Spaces' default). Secrets are injected at runtime — never baked in.

docker build -t vantage .
docker run --rm -p 7860:7860 --env-file .env vantage
# → http://localhost:7860

The databases are still external — the container expects an already-populated Qdrant and Neo4j via the .env credentials.

---

Project structure

packages/vantage_core/      # all business logic (hexagonal)
  domain/                   #   ports: rag, retrieval, gap, coverage, topic_index
  adapters/                 #   impls: qdrant, neo4j, openai, sentence-transformers, pytrends
  agent/                    #   LangGraph ReAct agent (graph.py, tools.py, trace.py)
  seeds/                    #   gartner_topics.csv (curated SSOT), topics.csv (generated)
apps/api/                   # FastAPI app + routers (chat, gaps, graph, signals)
apps/web/                   # React + Vite + Tailwind frontend
scripts/                    # data pipeline + dev CLIs (see below)
contracts/openapi.yaml      # generated API contract (scripts/export_openapi.py)
docs/                       # plans, ADRs

Scripts reference

Script	Purpose
healthcheck.py	Verify all external services are reachable
build_gartner_seeds.py	Generate topics.csv + topics_seed.json from the Gartner SSOT
mine_gdelt_topics.py	Augment topics with GDELT demand signals via BigQuery
download_data.sh	Fetch the HuffPost corpus into ./data
ingest.py / peek.py	Chunk+embed docs into Qdrant / verify the collection
tag_topics.py	Tag each chunk with the topics it mentions
ingest_topics.py	Embed the topic vocabulary into its own Qdrant collection
build_graph.py / wipe_graph.py	Build the Neo4j graph / reset it
gap_report.py · refresh_signals.py	Inspect gap scores / momentum signals
search.py · ask.py · agent_cli.py · demo_queries.py	Retrieval / RAG / agent CLIs
export_openapi.py	Regenerate contracts/openapi.yaml

---

Configuration

All runtime config is read from .env (validated at startup by vantage_core/config.py, fails fast on missing required vars). See .env.example for the full list. Notable knobs:

• OPENAI_MODEL — defaults to gpt-4o-mini
• INGEST_LIMIT — docs to ingest (default 2000; set 0 for the full ~200k corpus)
• LANGFUSE_* — optional; tracing is skipped if unset

---

⚠️ Windows users — read this

The data-ingestion pipeline cannot be run natively on Windows. Every orchestration path assumes a Unix shell and Unix tooling: download_data.sh and start.sh depend on bash, lsof, and curl; start.sh invokes a Unix conda path; and the Makefile targets call python3.11, which doesn't exist as a binary on Windows. Running the §Data pipeline or §Run the app sections from cmd/PowerShell will fail.

What Windows users can do: run the prebuilt Docker image (see §Run with Docker). That serves the full dashboard and agent against already-populated Qdrant + Neo4j databases. You can:

• ✅ browse the Knowledge Graph
• ✅ view the Gap Dashboard
• ✅ chat with the assistant (incl. live gap estimation)

What you cannot do from Windows: build, refresh, or re-ingest the corpus and graph — the ingestion scripts (ingest.py, tag_topics.py, build_graph.py, …) are pipeline-only and require the databases to be seeded by someone on macOS/Linux first. If you need to run the pipeline yourself, use macOS, Linux, or WSL2 — not a native Windows shell.