--- title: GAVL emoji: πŸ”¨ colorFrom: indigo colorTo: yellow sdk: docker app_port: 7860 pinned: false license: mit short_description: Cost-aware multi-agent LLM router with a live auction ---
# πŸ”¨ GAVL ### Ask more Β· Know faster Β· Pay less **A cost-aware, multi-agent LLM router.** Cheap models bid on every question, an auction picks a winner, a verifier checks the answer, and only the genuinely hard queries ever reach an expensive frontier model. Frontier-quality answers β€” without paying frontier prices on the easy 90%. ![Docker](https://img.shields.io/badge/deploy-docker-2496ED?logo=docker&logoColor=white) ![FastAPI](https://img.shields.io/badge/backend-FastAPI-009688?logo=fastapi&logoColor=white) ![Next.js](https://img.shields.io/badge/frontend-Next.js-000000?logo=nextdotjs&logoColor=white) ![OpenRouter](https://img.shields.io/badge/models-OpenRouter-6467F2) ![License](https://img.shields.io/badge/license-MIT-green)
--- ## The idea Most chat apps send **every** question to one big, expensive model β€” even "what's the capital of France." That's slow and wasteful: the easy majority of queries don't need a frontier model at all. GAVL treats routing as an **auction**. Three cheap, fast specialist models bid to answer each query based on how well they think they'd do. The best bid wins and drafts the answer, an independent verifier grades it, and *only* when a genuinely hard query fails does GAVL summon the expensive frontier model β€” the "boss fight." Easy questions never escalate. The payoff (see [evals](#-eval-results)): **~0.95 answer quality at ~68% lower cost and ~15Γ— faster median latency** than sending everything to the frontier. ## 🧠 How it works ```mermaid flowchart LR Q([Your query]) --> B{Parallel bids} B --> G[Generalist] B --> C[Coder] B --> M[Logic / Math] G & C & M --> A[Auction
score bids] A -->|winner drafts| V{Verifier} V -->|passes| βœ…([Answer]) V -->|hard + fails| BOSS[πŸ”¨ Frontier
boss fight] BOSS --> βœ… A -.needs fresh info.-> W[πŸ” Web search] W --> V ``` 1. **Bidding** β€” all three tier-1 models bid in parallel. Each returns a *confidence*, a *difficulty* estimate, and a flag for whether the query needs **live web data**. A confident bidder also drafts its answer on the spot, so if it wins there's no extra round-trip. 2. **Auction** β€” bids are scored on `0.7Β·confidence + 0.2Β·historical-accuracy βˆ’ 0.1Β·cost`. A **topic toggle** (general / coding / logic-math) lets you steer routing. The accuracy term is *learned* β€” a model that overbids and fails is trusted less next time. 3. **Verification** β€” an independent verifier grades the winning draft on correctness, completeness, and commitment. Creative writing skips this (no single right answer to check). 4. **Escalation** β€” only a **hard** query that fails verification escalates to the frontier model, with reasoning effort and token budget scaled to the difficulty. Easy queries never escalate; a weak answer just ships, clearly marked *unverified*. 5. **Web search** β€” if a bidder flags the query as needing current information (news, latest releases, "who won X", a specific recent item), the winner runs a live web search and **cites its sources**. ## ✨ Features - **Auction-based routing** with a learned per-model accuracy prior. - **Speculative drafting** β€” confident bidders answer inside their bid, so the winning answer often needs zero extra calls. - **Streaming-first UI** β€” you see text in ~3s; the verifier judges in parallel. - **Difficulty-gated escalation** β€” the frontier model is reserved for the small fraction of queries that truly need it. - **Live web search** with citations, gated on a per-query freshness flag. - **Topic toggles** to hint the router (general / coding / logic-math). - **Retro arcade UI** β€” a live "bidding bots" animation, a boss-fight ticker for escalations, per-code-block copy, and a `/explain` command that walks through the whole pipeline in-app. - **Cost & routing telemetry** β€” every answer shows who won, what it cost, and the verifier's score; a metrics dashboard tracks savings over time. ## πŸ“Š Eval results 38 bucketed queries (easy factual, subjective, typos, ambiguous, coding, medium reasoning, PhD-level STEM) run through the full pipeline vs. sending every query straight to the frontier model. Answers scored 0–1 by an independent LLM judge against reference notes. Frontier stand-in for the eval: DeepSeek R1; identical queries, models, and judge across both modes. | mode | judge score | tier-1 rate | p50 latency | total cost | |---|---|---|---|---| | **GAVL** | **0.95** | 89% | **3.6 s** | **$0.067** | | frontier-only | 1.00 | 0% | 52.8 s | $0.212 | **68% cheaper and ~15Γ— faster at the median**, giving up 0.05 judge points β€” half of which is a single eval-artifact failure (the frontier stand-in exhausted its token budget on one physics derivation), not a routing miss. At production frontier pricing the gap widens sharply, projecting to ~85–90% savings.
Per-bucket breakdown | bucket | n | judge | tier-1 | p50 | cost | |---|---|---|---|---|---| | easy_factual | 8 | 1.00 | 100% | 1.7 s | $0.002 | | subjective | 5 | 1.00 | 100% | 2.6 s | $0.002 | | typo | 4 | 1.00 | 100% | 3.3 s | $0.002 | | ambiguous | 5 | 0.82 | 100% | 2.8 s | $0.003 | | coding | 5 | 1.00 | 100% | 6.8 s | $0.005 | | reasoning | 5 | 1.00 | 100% | 9.9 s | $0.005 | | stem_hard | 6 | 0.83 | 33% | 112.6 s | $0.048 | Notably, 2 of 6 hard-STEM items were answered *correctly at tier 1* (judge 1.0, verifier-passed) β€” the cheap models legitimately solved them, so the "low" routing accuracy there is savings, not error.
Reproduce: ```bash cd backend FRONTIER_MODEL_ID=deepseek/deepseek-r1 uv run python -m evals.run_evals FRONTIER_MODEL_ID=deepseek/deepseek-r1 uv run python -m evals.run_evals --mode frontier ``` ## 🧩 Tech stack | Layer | Tech | |---|---| | Backend | FastAPI Β· LangGraph pipeline Β· async `httpx` | | Models | Routed through **OpenRouter** (swappable per slot via config) | | Frontend | Next.js (static export) Β· Tailwind Β· streaming NDJSON | | Store | MongoDB (optional) or in-memory | | Deploy | Single Docker image (Hugging Face Space) or split Vercel + HF | All model choices, auction weights, and thresholds live in `backend/app/config.py` β€” swap any bidder, the verifier, or the frontier model without touching pipeline code. ## πŸš€ Deployment The frontend is a pure client-side SPA and the backend is a pure API, so they can deploy independently: **Vercel** serves the UI, a **Hugging Face Docker Space** runs FastAPI. (The `Dockerfile` also bundles the UI, so the Space works standalone.) ### πŸ” Security model The API key is a server-side secret never sent to the browser β€” the real threat is *abuse of the endpoints that spend it*. Defense in depth, worst case bounded by a number: 1. **Credit-capped OpenRouter key** β€” a dedicated key with a hard credit limit. Provider-enforced; survives any app bug. 2. **Daily spend guard** β€” `DAILY_SPEND_LIMIT_USD`; query endpoints 503 once the day's total is exceeded. 3. **Access code** β€” every `/api/*` route requires the `X-Access-Code` header (`ACCESS_CODE`); `/health` stays open. Locking it also closes the run-history and metrics endpoints. 4. **Per-IP rate limiting** β€” `RATE_LIMIT_PER_MIN` / `_PER_DAY`. 5. **CORS** β€” `ALLOWED_ORIGINS` allowlist (browsers only; layers 1–4 are the real boundary). ### Backend β†’ Hugging Face Docker Space 1. Create the credit-capped OpenRouter key. 2. Push this repo to a **Docker** Space: ```bash git remote add hf https://huggingface.co/spaces//GAVL git push hf main:main ``` 3. **Settings β†’ Variables and secrets:** - `OPENROUTER_API_KEY` *(secret)* β€” the credit-capped key - `ACCESS_CODE` *(secret)* β€” the shared demo code - `TAVILY_API_KEY` *(secret, optional)* β€” enables the image strip under web-search answers. Omit it and the feature stays off; nothing else changes. - `ALLOWED_ORIGINS` *(variable)* β€” your Vercel URL (comma-separated) - `DAILY_SPEND_LIMIT_USD` *(variable)* β€” e.g. `20` - `MONGODB_URI` / `MONGODB_DB` *(optional)* β€” Atlas M0; else in-memory - `FRONTIER_MODEL_ID` *(optional)* 4. If using Atlas, allow `0.0.0.0/0` in its Network Access list. 5. Confirm `/health` returns `openrouter_key_set: true`. ### Frontend β†’ Vercel 1. Import `frontend/` as a Vercel project (auto-detected Next.js). 2. Set `NEXT_PUBLIC_API_BASE` = the HF Space URL. (The access code is entered at runtime, never baked into the bundle.) 3. Deploy, add the Vercel domain to the Space's `ALLOWED_ORIGINS`, redeploy. > HF free tier sleeps after ~48h idle β†’ first query cold-starts ~30s. HF Pro or > an always-on backend (Fly.io / Render, same Dockerfile) removes this. CPU > Basic is sufficient β€” all inference happens on OpenRouter. ## πŸ’» Local development ```bash # backend cd backend && uv sync && uv run uvicorn app.main:app --reload --port 8000 # frontend cd frontend && npm install && npm run dev ``` Then open . With no `ACCESS_CODE` set the gate is skipped automatically; set one in `backend/.env` to preview the splash screen. ---
Built with FastAPI, LangGraph, Next.js, and OpenRouter Β· MIT licensed