--- title: Vantage emoji: πŸ“Š colorFrom: blue colorTo: indigo sdk: docker app_port: 7860 pinned: false --- # Vantage β€” Agentic Market Intelligence Vantage is an agentic RAG system that surfaces **emerging content gaps** β€” topics the market is demanding but your corpus barely covers. It fuses live demand signals with vector coverage and a knowledge graph, then lets a LangGraph agent reason over all three. - **Google Trends signals** β€” demand volume + velocity per topic - **Qdrant** β€” hybrid (dense + sparse) vector search over a ~200k-doc corpus - **Neo4j** β€” knowledge graph of `(:Topic)-[:COVERED_BY]->(:Document)` edges - **LangGraph agent** β€” orchestrates gap analysis, RAG search, and signals with multi-turn memory **Gap score** = `Demand Γ— (1 βˆ’ Coverage)` β€” high demand, low coverage β‡’ a gap worth filling. --- ## Architecture Vantage runs as two planes that share the same core library and the same two databases: an **offline pipeline** that *populates* Qdrant + Neo4j, and an **online service** that only *reads* them. The app never writes corpus data at request time β€” it serves what the pipeline already built. ### Serving plane (what runs at request time) ```mermaid flowchart LR User([Browser]) --> Web[React + Vite UI] Web -->|REST /api| API[FastAPI
chat Β· gaps Β· graph Β· signals] API --> Agent[LangGraph ReAct agent] Agent -->|rag_search| RAG[Hybrid retrieval
+ rerank] Agent -->|gap_analysis Β· trending_topics
coverage_lookup Β· graph_lookup| GRAPH[(Neo4j
Topic→Document)] Agent -->|estimate_gap| LIVE[Live Trends + coverage] RAG --> QDRANT[(Qdrant
dense + sparse)] RAG --> OPENAI{{OpenAI
gpt-4o-mini}} LIVE --> TRENDS{{Google Trends}} LIVE --> QDRANT ``` **Request lifecycle:** the UI calls a FastAPI route β†’ the agent picks tools from their docstrings β†’ each tool runs through a **domain port** (`rag`, `retrieval`, `gap`, `coverage`, `topic_index`) backed by an **adapter** (Qdrant, Neo4j, OpenAI, sentence-transformers, pytrends) β†’ the agent synthesises a grounded answer, citing sources and surfacing the numbers. ### Ingestion plane (offline β€” run once, before serving) ```mermaid flowchart LR SEEDS[build_gartner_seeds
+ mine_gdelt_topics] --> TOPICS[(topic vocab)] CORPUS[download_data] --> INGEST[ingest.py
chunk + embed] INGEST --> QDRANT[(Qdrant)] TOPICS --> TAG[tag_topics] --> COV[topic_coverage.csv] TOPICS --> TIDX[ingest_topics] --> QDRANT COV --> BUILD[build_graph
coverage Γ— signals Γ— gap] --> NEO[(Neo4j)] ``` This plane is **bash/Unix-only** and is what Windows users cannot run (see the Windows note at the end). ### Layering All business logic lives in `packages/vantage_core` and follows a **hexagonal** design: | Layer | Location | Responsibility | |-------|----------|----------------| | **Ports** (domain) | `vantage_core/domain/` | What the app does β€” retrieval, RAG, gap & coverage scoring β€” independent of any vendor | | **Adapters** | `vantage_core/adapters/` | How it's done β€” Qdrant, Neo4j, OpenAI, sentence-transformers, pytrends | | **Agent** | `vantage_core/agent/` | LangGraph ReAct loop + the six tools that wrap the ports | | **Entry points** | `apps/api`, `scripts/` | Thin shells: the HTTP API and the pipeline/dev CLIs | --- ## Prerequisites You need accounts / endpoints for four external services (all have free tiers): | Service | Used for | Required | |----------------|---------------------------------|----------| | Qdrant Cloud | Vector search | βœ… | | Neo4j AuraDB | Knowledge graph | βœ… | | OpenAI | LLM (default `gpt-4o-mini`) | βœ… | | Langfuse Cloud | LLM tracing | optional | Plus: **Python 3.11**, **Node 20+**, and (optionally) **Docker**. --- ## Setup ```bash # 1. Create the Python env (conda) and install the package + API deps conda env create -f environment.yaml # creates the 'py311' env conda activate py311 make install # pip install -e vantage_core + API reqs # 2. Configure credentials cp .env.example .env $EDITOR .env # fill in Qdrant / Neo4j / OpenAI keys # 3. Verify every external service is reachable (pass/fail gate) make healthcheck ``` `make healthcheck` pings Qdrant, Neo4j, OpenAI, and Langfuse and prints a colour-coded status line for each. Don't proceed until it's green. --- ## Data pipeline β€” run this once, in order This populates Qdrant and Neo4j. Every step is **idempotent** (safe to re-run). The app won't return anything until these have run, because `start.sh` only *serves* what's already in the databases. ```bash # ── 1. TOPICS β€” build the topic vocabulary (the "get topics" step) ──────────── python scripts/build_gartner_seeds.py # β†’ seeds/topics.csv + data/topics_seed.json python scripts/mine_gdelt_topics.py # augment with fresh GDELT demand-side topics (BigQuery) # ── 2. DATA β€” fetch the HuffPost corpus into ./data (gitignored) ────────────── bash scripts/download_data.sh # see script header for auth options # ── 3. INGEST β€” chunk + embed docs β†’ Qdrant ────────────────────────────────── python scripts/ingest.py # honours INGEST_LIMIT (default 2000; 0 = full) python scripts/peek.py # verify: point count + sample payload # ── 4. TAG β€” label each chunk with the topics it mentions ───────────────────── python scripts/tag_topics.py # β†’ topic_coverage.csv (read by step 6) # ── 5. TOPIC INDEX β€” embed the vocabulary into its own collection ───────────── python scripts/ingest_topics.py # β†’ vantage_topics collection (semantic gap match) # ── 6. GRAPH β€” coverage Γ— signals Γ— gap score β†’ Neo4j ───────────────────────── python scripts/build_graph.py # builds (:Topic)-[:COVERED_BY]->(:Document) ``` > **Order matters at two points:** topics (1) must exist before tagging (4) and the topic > index (5); the graph (6) reads `topic_coverage.csv` from tagging (4). Everything else is > independent. To start the graph over, run `python scripts/wipe_graph.py` first. Quick sanity checks (read-only, no writes): ```bash python scripts/gap_report.py # ranked GapScore table python scripts/search.py "your query" # hybrid retrieval + rerank python scripts/ask.py "your question" # full RAG answer with citations python scripts/agent_cli.py # interactive agent REPL python scripts/demo_queries.py # 4 canned end-to-end agent queries ``` --- ## Run the app Once the pipeline above has populated the databases, start the services. **This is the last step.** ```bash make start # or: bash start.sh ``` - **API** β†’ http://localhost:8000 (FastAPI + agent; logs in `api.log`) - **Web** β†’ http://localhost:5173 (React/Vite UI; logs in `web.log`) `start.sh` stops anything stale on ports 8000/5173, launches the API, polls its healthcheck until ready, then launches the frontend. To stop everything: ```bash make stop # or: bash stop.sh ``` > **Note:** `start.sh` currently invokes Python via a hard-coded conda path > (`/Users/sourabh/opt/miniconda3/envs/py311/bin/python`). If your env lives elsewhere, > edit that line or run uvicorn directly: > `python -m uvicorn apps.api.main:app --port 8000 --reload`. --- ## Run with Docker (single container) The `Dockerfile` builds the frontend, bakes the ML models in for offline inference, and serves the API + static UI from one image on port **7860** (Hugging Face Spaces' default). Secrets are injected at runtime β€” never baked in. ```bash docker build -t vantage . docker run --rm -p 7860:7860 --env-file .env vantage # β†’ http://localhost:7860 ``` The databases are still external β€” the container expects an already-populated Qdrant and Neo4j via the `.env` credentials. --- ## Project structure ``` packages/vantage_core/ # all business logic (hexagonal) domain/ # ports: rag, retrieval, gap, coverage, topic_index adapters/ # impls: qdrant, neo4j, openai, sentence-transformers, pytrends agent/ # LangGraph ReAct agent (graph.py, tools.py, trace.py) seeds/ # gartner_topics.csv (curated SSOT), topics.csv (generated) apps/api/ # FastAPI app + routers (chat, gaps, graph, signals) apps/web/ # React + Vite + Tailwind frontend scripts/ # data pipeline + dev CLIs (see below) contracts/openapi.yaml # generated API contract (scripts/export_openapi.py) docs/ # plans, ADRs ``` ### Scripts reference | Script | Purpose | |--------|---------| | `healthcheck.py` | Verify all external services are reachable | | `build_gartner_seeds.py` | Generate `topics.csv` + `topics_seed.json` from the Gartner SSOT | | `mine_gdelt_topics.py` | Augment topics with GDELT demand signals via BigQuery | | `download_data.sh` | Fetch the HuffPost corpus into `./data` | | `ingest.py` / `peek.py` | Chunk+embed docs into Qdrant / verify the collection | | `tag_topics.py` | Tag each chunk with the topics it mentions | | `ingest_topics.py` | Embed the topic vocabulary into its own Qdrant collection | | `build_graph.py` / `wipe_graph.py` | Build the Neo4j graph / reset it | | `gap_report.py` Β· `refresh_signals.py` | Inspect gap scores / momentum signals | | `search.py` Β· `ask.py` Β· `agent_cli.py` Β· `demo_queries.py` | Retrieval / RAG / agent CLIs | | `export_openapi.py` | Regenerate `contracts/openapi.yaml` | --- ## Configuration All runtime config is read from `.env` (validated at startup by `vantage_core/config.py`, fails fast on missing required vars). See `.env.example` for the full list. Notable knobs: - `OPENAI_MODEL` β€” defaults to `gpt-4o-mini` - `INGEST_LIMIT` β€” docs to ingest (default `2000`; set `0` for the full ~200k corpus) - `LANGFUSE_*` β€” optional; tracing is skipped if unset --- ## ⚠️ Windows users β€” read this The **data-ingestion pipeline cannot be run natively on Windows.** Every orchestration path assumes a Unix shell and Unix tooling: `download_data.sh` and `start.sh` depend on `bash`, `lsof`, and `curl`; `start.sh` invokes a Unix conda path; and the Makefile targets call `python3.11`, which doesn't exist as a binary on Windows. Running the Β§Data pipeline or Β§Run the app sections from `cmd`/PowerShell will fail. **What Windows users *can* do:** run the prebuilt **Docker image** (see Β§Run with Docker). That serves the full dashboard and agent against **already-populated** Qdrant + Neo4j databases. You can: - βœ… browse the **Knowledge Graph** - βœ… view the **Gap Dashboard** - βœ… chat with the **assistant** (incl. live gap estimation) **What you cannot do from Windows:** build, refresh, or re-ingest the corpus and graph β€” the ingestion scripts (`ingest.py`, `tag_topics.py`, `build_graph.py`, …) are pipeline-only and require the databases to be seeded by someone on macOS/Linux first. If you need to run the pipeline yourself, use **macOS, Linux, or WSL2** β€” not a native Windows shell.