Spaces:
Runtime error
Runtime error
| title: Vantage | |
| emoji: 📊 | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # Vantage — Agentic Market Intelligence | |
| Vantage is an agentic RAG system that surfaces **emerging content gaps** — topics the | |
| market is demanding but your corpus barely covers. It fuses live demand signals with | |
| vector coverage and a knowledge graph, then lets a LangGraph agent reason over all three. | |
| - **Google Trends signals** — demand volume + velocity per topic | |
| - **Qdrant** — hybrid (dense + sparse) vector search over a ~200k-doc corpus | |
| - **Neo4j** — knowledge graph of `(:Topic)-[:COVERED_BY]->(:Document)` edges | |
| - **LangGraph agent** — orchestrates gap analysis, RAG search, and signals with multi-turn memory | |
| **Gap score** = `Demand × (1 − Coverage)` — high demand, low coverage ⇒ a gap worth filling. | |
| --- | |
| ## Architecture | |
| Vantage runs as two planes that share the same core library and the same two databases: | |
| an **offline pipeline** that *populates* Qdrant + Neo4j, and an **online service** that only | |
| *reads* them. The app never writes corpus data at request time — it serves what the pipeline | |
| already built. | |
| ### Serving plane (what runs at request time) | |
| ```mermaid | |
| flowchart LR | |
| User([Browser]) --> Web[React + Vite UI] | |
| Web -->|REST /api| API[FastAPI<br/>chat · gaps · graph · signals] | |
| API --> Agent[LangGraph ReAct agent] | |
| Agent -->|rag_search| RAG[Hybrid retrieval<br/>+ rerank] | |
| Agent -->|gap_analysis · trending_topics<br/>coverage_lookup · graph_lookup| GRAPH[(Neo4j<br/>Topic→Document)] | |
| Agent -->|estimate_gap| LIVE[Live Trends + coverage] | |
| RAG --> QDRANT[(Qdrant<br/>dense + sparse)] | |
| RAG --> OPENAI{{OpenAI<br/>gpt-4o-mini}} | |
| LIVE --> TRENDS{{Google Trends}} | |
| LIVE --> QDRANT | |
| ``` | |
| **Request lifecycle:** the UI calls a FastAPI route → the agent picks tools from their | |
| docstrings → each tool runs through a **domain port** (`rag`, `retrieval`, `gap`, `coverage`, | |
| `topic_index`) backed by an **adapter** (Qdrant, Neo4j, OpenAI, sentence-transformers, | |
| pytrends) → the agent synthesises a grounded answer, citing sources and surfacing the numbers. | |
| ### Ingestion plane (offline — run once, before serving) | |
| ```mermaid | |
| flowchart LR | |
| SEEDS[build_gartner_seeds<br/>+ mine_gdelt_topics] --> TOPICS[(topic vocab)] | |
| CORPUS[download_data] --> INGEST[ingest.py<br/>chunk + embed] | |
| INGEST --> QDRANT[(Qdrant)] | |
| TOPICS --> TAG[tag_topics] --> COV[topic_coverage.csv] | |
| TOPICS --> TIDX[ingest_topics] --> QDRANT | |
| COV --> BUILD[build_graph<br/>coverage × signals × gap] --> NEO[(Neo4j)] | |
| ``` | |
| This plane is **bash/Unix-only** and is what Windows users cannot run (see the Windows note at the end). | |
| ### Layering | |
| All business logic lives in `packages/vantage_core` and follows a **hexagonal** design: | |
| | Layer | Location | Responsibility | | |
| |-------|----------|----------------| | |
| | **Ports** (domain) | `vantage_core/domain/` | What the app does — retrieval, RAG, gap & coverage scoring — independent of any vendor | | |
| | **Adapters** | `vantage_core/adapters/` | How it's done — Qdrant, Neo4j, OpenAI, sentence-transformers, pytrends | | |
| | **Agent** | `vantage_core/agent/` | LangGraph ReAct loop + the six tools that wrap the ports | | |
| | **Entry points** | `apps/api`, `scripts/` | Thin shells: the HTTP API and the pipeline/dev CLIs | | |
| --- | |
| ## Prerequisites | |
| You need accounts / endpoints for four external services (all have free tiers): | |
| | Service | Used for | Required | | |
| |----------------|---------------------------------|----------| | |
| | Qdrant Cloud | Vector search | ✅ | | |
| | Neo4j AuraDB | Knowledge graph | ✅ | | |
| | OpenAI | LLM (default `gpt-4o-mini`) | ✅ | | |
| | Langfuse Cloud | LLM tracing | optional | | |
| Plus: **Python 3.11**, **Node 20+**, and (optionally) **Docker**. | |
| --- | |
| ## Setup | |
| ```bash | |
| # 1. Create the Python env (conda) and install the package + API deps | |
| conda env create -f environment.yaml # creates the 'py311' env | |
| conda activate py311 | |
| make install # pip install -e vantage_core + API reqs | |
| # 2. Configure credentials | |
| cp .env.example .env | |
| $EDITOR .env # fill in Qdrant / Neo4j / OpenAI keys | |
| # 3. Verify every external service is reachable (pass/fail gate) | |
| make healthcheck | |
| ``` | |
| `make healthcheck` pings Qdrant, Neo4j, OpenAI, and Langfuse and prints a colour-coded | |
| status line for each. Don't proceed until it's green. | |
| --- | |
| ## Data pipeline — run this once, in order | |
| This populates Qdrant and Neo4j. Every step is **idempotent** (safe to re-run). The app | |
| won't return anything until these have run, because `start.sh` only *serves* what's | |
| already in the databases. | |
| ```bash | |
| # ── 1. TOPICS — build the topic vocabulary (the "get topics" step) ──────────── | |
| python scripts/build_gartner_seeds.py # → seeds/topics.csv + data/topics_seed.json | |
| python scripts/mine_gdelt_topics.py # augment with fresh GDELT demand-side topics (BigQuery) | |
| # ── 2. DATA — fetch the HuffPost corpus into ./data (gitignored) ────────────── | |
| bash scripts/download_data.sh # see script header for auth options | |
| # ── 3. INGEST — chunk + embed docs → Qdrant ────────────────────────────────── | |
| python scripts/ingest.py # honours INGEST_LIMIT (default 2000; 0 = full) | |
| python scripts/peek.py # verify: point count + sample payload | |
| # ── 4. TAG — label each chunk with the topics it mentions ───────────────────── | |
| python scripts/tag_topics.py # → topic_coverage.csv (read by step 6) | |
| # ── 5. TOPIC INDEX — embed the vocabulary into its own collection ───────────── | |
| python scripts/ingest_topics.py # → vantage_topics collection (semantic gap match) | |
| # ── 6. GRAPH — coverage × signals × gap score → Neo4j ───────────────────────── | |
| python scripts/build_graph.py # builds (:Topic)-[:COVERED_BY]->(:Document) | |
| ``` | |
| > **Order matters at two points:** topics (1) must exist before tagging (4) and the topic | |
| > index (5); the graph (6) reads `topic_coverage.csv` from tagging (4). Everything else is | |
| > independent. To start the graph over, run `python scripts/wipe_graph.py` first. | |
| Quick sanity checks (read-only, no writes): | |
| ```bash | |
| python scripts/gap_report.py # ranked GapScore table | |
| python scripts/search.py "your query" # hybrid retrieval + rerank | |
| python scripts/ask.py "your question" # full RAG answer with citations | |
| python scripts/agent_cli.py # interactive agent REPL | |
| python scripts/demo_queries.py # 4 canned end-to-end agent queries | |
| ``` | |
| --- | |
| ## Run the app | |
| Once the pipeline above has populated the databases, start the services. **This is the | |
| last step.** | |
| ```bash | |
| make start # or: bash start.sh | |
| ``` | |
| - **API** → http://localhost:8000 (FastAPI + agent; logs in `api.log`) | |
| - **Web** → http://localhost:5173 (React/Vite UI; logs in `web.log`) | |
| `start.sh` stops anything stale on ports 8000/5173, launches the API, polls its | |
| healthcheck until ready, then launches the frontend. To stop everything: | |
| ```bash | |
| make stop # or: bash stop.sh | |
| ``` | |
| > **Note:** `start.sh` currently invokes Python via a hard-coded conda path | |
| > (`/Users/sourabh/opt/miniconda3/envs/py311/bin/python`). If your env lives elsewhere, | |
| > edit that line or run uvicorn directly: | |
| > `python -m uvicorn apps.api.main:app --port 8000 --reload`. | |
| --- | |
| ## Run with Docker (single container) | |
| The `Dockerfile` builds the frontend, bakes the ML models in for offline inference, and | |
| serves the API + static UI from one image on port **7860** (Hugging Face Spaces' default). | |
| Secrets are injected at runtime — never baked in. | |
| ```bash | |
| docker build -t vantage . | |
| docker run --rm -p 7860:7860 --env-file .env vantage | |
| # → http://localhost:7860 | |
| ``` | |
| The databases are still external — the container expects an already-populated Qdrant and | |
| Neo4j via the `.env` credentials. | |
| --- | |
| ## Project structure | |
| ``` | |
| packages/vantage_core/ # all business logic (hexagonal) | |
| domain/ # ports: rag, retrieval, gap, coverage, topic_index | |
| adapters/ # impls: qdrant, neo4j, openai, sentence-transformers, pytrends | |
| agent/ # LangGraph ReAct agent (graph.py, tools.py, trace.py) | |
| seeds/ # gartner_topics.csv (curated SSOT), topics.csv (generated) | |
| apps/api/ # FastAPI app + routers (chat, gaps, graph, signals) | |
| apps/web/ # React + Vite + Tailwind frontend | |
| scripts/ # data pipeline + dev CLIs (see below) | |
| contracts/openapi.yaml # generated API contract (scripts/export_openapi.py) | |
| docs/ # plans, ADRs | |
| ``` | |
| ### Scripts reference | |
| | Script | Purpose | | |
| |--------|---------| | |
| | `healthcheck.py` | Verify all external services are reachable | | |
| | `build_gartner_seeds.py` | Generate `topics.csv` + `topics_seed.json` from the Gartner SSOT | | |
| | `mine_gdelt_topics.py` | Augment topics with GDELT demand signals via BigQuery | | |
| | `download_data.sh` | Fetch the HuffPost corpus into `./data` | | |
| | `ingest.py` / `peek.py` | Chunk+embed docs into Qdrant / verify the collection | | |
| | `tag_topics.py` | Tag each chunk with the topics it mentions | | |
| | `ingest_topics.py` | Embed the topic vocabulary into its own Qdrant collection | | |
| | `build_graph.py` / `wipe_graph.py` | Build the Neo4j graph / reset it | | |
| | `gap_report.py` · `refresh_signals.py` | Inspect gap scores / momentum signals | | |
| | `search.py` · `ask.py` · `agent_cli.py` · `demo_queries.py` | Retrieval / RAG / agent CLIs | | |
| | `export_openapi.py` | Regenerate `contracts/openapi.yaml` | | |
| --- | |
| ## Configuration | |
| All runtime config is read from `.env` (validated at startup by `vantage_core/config.py`, | |
| fails fast on missing required vars). See `.env.example` for the full list. Notable knobs: | |
| - `OPENAI_MODEL` — defaults to `gpt-4o-mini` | |
| - `INGEST_LIMIT` — docs to ingest (default `2000`; set `0` for the full ~200k corpus) | |
| - `LANGFUSE_*` — optional; tracing is skipped if unset | |
| --- | |
| ## ⚠️ Windows users — read this | |
| The **data-ingestion pipeline cannot be run natively on Windows.** Every orchestration | |
| path assumes a Unix shell and Unix tooling: `download_data.sh` and `start.sh` depend on | |
| `bash`, `lsof`, and `curl`; `start.sh` invokes a Unix conda path; and the Makefile targets | |
| call `python3.11`, which doesn't exist as a binary on Windows. Running the §Data pipeline or | |
| §Run the app sections from `cmd`/PowerShell will fail. | |
| **What Windows users *can* do:** run the prebuilt **Docker image** (see §Run with Docker). | |
| That serves the full dashboard and agent against **already-populated** Qdrant + Neo4j | |
| databases. You can: | |
| - ✅ browse the **Knowledge Graph** | |
| - ✅ view the **Gap Dashboard** | |
| - ✅ chat with the **assistant** (incl. live gap estimation) | |
| **What you cannot do from Windows:** build, refresh, or re-ingest the corpus and graph — | |
| the ingestion scripts (`ingest.py`, `tag_topics.py`, `build_graph.py`, …) are pipeline-only | |
| and require the databases to be seeded by someone on macOS/Linux first. If you need to run | |
| the pipeline yourself, use **macOS, Linux, or WSL2** — not a native Windows shell. | |