| --- |
| language: |
| - en |
| license: mit |
| library_name: llama-cpp-python |
| base_model: katanemo/Arch-Router-1.5B |
| tags: |
| - routing |
| - grpo |
| - reinforcement-learning |
| - gguf |
| - lora |
| - unsloth |
| - trl |
| - qwen2 |
| - llama-cpp |
| - contract-aware |
| - cost-optimization |
| - query-classification |
| model_type: qwen2 |
| pipeline_tag: text-classification |
| datasets: |
| - custom |
| metrics: |
| - accuracy |
| model-index: |
| - name: ModelGate-Router |
| results: |
| - task: |
| type: text-classification |
| name: Query Complexity Classification |
| metrics: |
| - type: accuracy |
| value: 83.3 |
| name: Overall Accuracy (held-out, GGUF Q8_0) |
| - type: accuracy |
| value: 85.7 |
| name: Medium Tier Accuracy |
| - type: latency |
| value: 62 |
| name: Avg Latency (ms, CUDA) |
| --- |
| |
| <p align="center"> |
| <img src="banner.svg" alt="ModelGate Banner" width="100%"/> |
| </p> |
|
|
| # ModelGate |
|
|
| **Intelligent AI Routing - Built from Your Contracts** |
|
|
| One line of code changed. Millions of premium calls rerouted. |
|
|
| ModelGate is a contract-aware AI control plane that ingests customer contracts, extracts SLA/privacy/routing constraints, and generates an OpenAI-compatible endpoint that automatically routes every request to the optimal model. Simple queries go to cheap models. Complex queries go to premium ones. Contract compliance is enforced per request, automatically. |
|
|
| **3rd Place** at the KSU Social Good Hackathon 2026 - Assurant Track. |
|
|
| ### Team Agents Assemble |
|
|
| | | Role | |
| |---|---| |
| | **[Aaryan Kapoor](https://www.linkedin.com/in/theaaryankapoor/)** | Lead Architect & AI Engineer | |
| | **[Pradyumna Kumar](https://www.linkedin.com/in/pradyum-kumar/)** | Platform Architect & Frontend | |
| | **[Danny Tran](https://www.linkedin.com/in/nam-tr%E1%BA%A7n-02973b2b6/)** | Design & Presentation Lead | |
|
|
| ## Why |
|
|
| Over 30 new LLMs launched in the past month alone. No team has time to evaluate them all - so they pick one premium model and send everything to it. The result: 50-90% of enterprise AI spend is wasted on over-provisioned models, and premium models consume 180x more energy per query than small ones. |
|
|
| ModelGate fixes this. You change one line of code - your `base_url` - and we handle model selection, contract compliance, and cost optimization automatically. |
|
|
| ## Results |
|
|
| ### MMLU Routing Benchmark (60 questions, 6 subjects) |
|
|
| We benchmarked ModelGate against always routing to GPT-5.4 (default reasoning): |
|
|
| | | GPT-5.4 Direct | ModelGate Router | Delta | |
| |---|---|---|---| |
| | **Overall Accuracy** | 90% | 85% | -5pp | |
| | **Hard Accuracy** | 80% | 80% | 0 | |
| | **Cost** | $0.023 | $0.0095 | **-59%** | |
|
|
| The router sent 68% of queries to Gemini Flash Lite, 17% to GPT-4o-mini, and only 15% to GPT-5.4. Hard questions were routed correctly - the cost savings come from not overpaying on easy ones. |
|
|
| Projected at 10k requests/month: **$1.58 vs $3.83** (59% savings). |
|
|
| ### Fine-Tuned Classification Model (GRPO Reinforcement Learning) |
|
|
| We fine-tuned ModelGate-Router (based on Arch-Router-1.5B) using GRPO to fix a critical blind spot: the stock model misclassified 86% of medium-complexity queries as complex. |
|
|
| | Tier | Stock | Fine-Tuned | Delta | |
| |---|---|---|---| |
| | Simple | 87.9% | 81.8% | -6.1pp | |
| | **Medium** | **14.3%** | **85.7%** | **+71.4pp** | |
| | Complex | 100% | 85.7% | -14.3pp | |
| | **Overall** | **70.4%** | **83.3%** | **+13.0pp** | |
|
|
| - **Training:** 2.5 minutes, 150 steps, 172 labeled prompts, LoRA rank 32 (2.3% of params) |
| - **Hardware:** RTX 3080 Laptop, 8GB VRAM |
| - **Inference:** GGUF Q8_0 quantized to 1.6 GB, runs at **62ms** per classification (3.2x faster than FP16) |
| - **Eval:** 54 held-out prompts, zero overlap with training data |
| - **Download:** [ModelGate-Router on HuggingFace](https://huggingface.co/AaryanK/ModelGate) |
| |
| ## Screenshots |
| |
| ### Platform Dashboard |
| Real-time monitoring of routing decisions, cost savings, model distribution, and request volume. |
| |
| <p align="center"> |
| <img src="screenshot.png" alt="ModelGate Dashboard" width="100%"/> |
| </p> |
| |
| ### Model Registry |
| Browse the OpenRouter catalog and toggle models on/off with one click. Configure which models power your routing. |
| |
| <p align="center"> |
| <img src="screenshot_2.png" alt="Model Registry" width="100%"/> |
| </p> |
| |
| ### Customer Onboarding |
| Upload a contract, review the AI-extracted profile, and start routing - all in under 30 seconds. |
| |
| <p align="center"> |
| <img src="screenshot_3.png" alt="Customer Onboarding" width="100%"/> |
| </p> |
| |
| ## How It Works |
| |
| ``` |
| Contract (PDF/text) → LLM extracts constraints → Customer AI Profile → OpenAI-compatible endpoint |
| ↓ |
| Prompt received |
| ↓ |
| ModelGate-Router classifies |
| (simple / medium / complex) |
| ↓ |
| Route to optimal model |
| per contract constraints |
| ``` |
| |
| 1. **Upload** a customer contract (SLA, privacy docs, compliance requirements) |
| 2. **Extract** - an LLM analyzes the contract and produces a structured customer profile (region restrictions, allowed providers, latency targets, cost sensitivity) |
| 3. **Route** - each request is classified by the fine-tuned 1.5B router (~62ms) and sent to the cheapest model that satisfies all contract constraints |
| 4. **Monitor** - dashboard shows routing decisions, model distribution, cost savings, and per-request traces |
| |
| ## Architecture |
| |
| ``` |
| [Next.js Dashboard :3000] → [FastAPI :8000] → [OpenRouter / Direct APIs] |
| ↓ |
| [ModelGate-Router GGUF] |
| (llama.cpp, CUDA, ~62ms) |
| ``` |
| |
| | Component | Stack | |
| |---|---| |
| | Backend | Python, FastAPI, SQLite | |
| | Frontend | Next.js 16, TypeScript, Tailwind CSS, shadcn/ui, Recharts | |
| | Classification | ModelGate-Router (fine-tuned), GGUF Q8_0, llama-cpp-python | |
| | LLM Inference | OpenRouter (multi-provider: OpenAI, Google, Anthropic, etc.) | |
| | Contract Extraction | LLM-powered (GPT-5.4) | |
|
|
| ## Quick Start |
|
|
| ### Prerequisites |
| - Python 3.12 with PyTorch + CUDA |
| - Node.js 18+ |
| - NVIDIA GPU (for classification model) |
| - OpenRouter API key |
|
|
| ### Setup |
|
|
| ```bash |
| git clone https://github.com/Aaryan-Kapoor/ModelGate-Hackathon |
| cd ModelGate-Hackathon |
| |
| # Add your API key |
| cp .env.example .env |
| # Edit .env with your OPENROUTER_API_KEY |
| |
| # Run everything |
| chmod +x scripts/start.sh |
| ./scripts/start.sh |
| ``` |
|
|
| Or manually: |
|
|
| ```bash |
| # Backend |
| python3.12 -m venv backend/venv --system-site-packages |
| source backend/venv/bin/activate |
| pip install -r backend/requirements.txt |
| python scripts/seed_data.py |
| uvicorn backend.main:app --port 8000 |
| |
| # Frontend (separate terminal) |
| cd frontend && npm install && npm run dev |
| ``` |
|
|
| ### Access |
| - Dashboard: http://localhost:3000 |
| - API Docs: http://localhost:8000/docs |
| - Proxy endpoint: `POST http://localhost:8000/v1/{customer_id}/chat/completions` |
|
|
| ## Benchmarking |
|
|
| ```bash |
| # Run MMLU benchmark against any OpenAI-compatible endpoint |
| python scripts/bench_mmlu.py run \ |
| --base-url http://localhost:8000/v1 \ |
| --api-key dummy \ |
| --model auto \ |
| --label router |
| |
| # Compare two runs |
| python scripts/bench_mmlu.py compare results/run_a.json results/run_b.json |
| |
| # Benchmark the classification model (GGUF) |
| python finetuning/bench_gguf.py |
| ``` |
|
|
| ## Fine-Tuning |
|
|
| The fine-tuning pipeline lives in `finetuning/`. See [`finetuning/README.md`](finetuning/README.md) for full details. |
|
|
| ```bash |
| # Train (2.5 min on RTX 3080 8GB) |
| python finetuning/grpo_run_nocot.py |
| |
| # Export to GGUF |
| python finetuning/export_gguf.py nocot |
| |
| # Benchmark stock vs fine-tuned |
| python finetuning/bench_gguf.py |
| ``` |
|
|
| ## Project Structure |
|
|
| ``` |
| backend/ |
| main.py # FastAPI app |
| services/ |
| classifier.py # ModelGate-Router inference (llama.cpp) |
| extractor.py # Contract → Customer AI Profile (LLM) |
| router_engine.py # Model scoring and selection |
| provider_registry.py # Model catalog with pricing/capabilities |
| frontend/ # Next.js dashboard |
| finetuning/ |
| grpo_run_nocot.py # GRPO training script |
| grpo_training_data.json # 172 labeled training prompts |
| grpo_eval_data.json # 54 held-out eval prompts |
| export_gguf.py # LoRA merge + GGUF conversion |
| bench_gguf.py # GGUF benchmark (accuracy + latency) |
| ModelGate-Router.Q8_0.gguf # Production model (1.6 GB) |
| scripts/ |
| bench_mmlu.py # MMLU benchmark runner |
| mmlu_questions.json # 60 real MMLU questions from HuggingFace |
| start.sh # One-command startup |
| ``` |
