Benchgen Router Lite (head-Qwen3-1.7B-e99b449fde)
Repo: benchgen/benchgen-router-lite · Built by Benchgen, the benchmarking infrastructure for AI agents.
A router head, not a fine-tuned LLM: a tiny (10,245-parameter) linear classifier that sits on top of a frozen Qwen/Qwen3-1.7B backbone and learns which model in a pool should answer a given query. The backbone never updates, only this head does. It is trained with separable CMA-ES (a gradient-free evolutionary search) against a measured reward matrix, not gradient descent, and it never calls any pool model during training.
Full write-up (dataset, training run, benchmark results): Router Lite guide.
Intended use
- Research and teaching on LLM routing: how a small head can pick among a fixed pool of models.
- A starting point you can retrain on your own pool and your own reward matrix.
- A reproducible example of evaluating a router on Benchgen (see "Run the same benchmark" below).
Out of scope: production routing without retraining on your own pool and data, any language other than English, and domains outside math, knowledge and reasoning.
How it works
query --> frozen Qwen3-1.7B (mean-pooled embedding, 2048-d)
--> this head (linear: logits = embedding @ W + b)
--> argmax over 5 pool agents
--> picked agent's REAL reply is what gets returned
The router itself never generates an answer. It only picks who should.
Pool (5 agents this head was trained to route between)
| Slot | Agent id (agent_order) |
Underlying model | Role |
|---|---|---|---|
| Frontier | frontier_a |
openai/gpt-oss-120b |
Reasoning |
| Frontier | frontier_b |
deepseek/deepseek-v4-flash-0731 |
Reasoning |
| Frontier | frontier_c |
google/gemma-3-27b-it |
Direct |
| Mid-tier | open_mid |
mistralai/mistral-nemo |
Direct (deliberately the weakest agent) |
| Cheap reasoning | open_cheap_reasoning |
inclusionai/ling-3.0-flash |
Reasoning |
The underlying model slugs are not baked into the head. They are the OpenRouter pool this specific run was trained against. A caller resolves picked_agent to a real model separately (the Benchgen gateway does this through its own pool config).
Files
| File | What it is |
|---|---|
head_weights.npy |
Flat float32 parameter vector (10,245 values), theta |
manifest.json |
Architecture and training metadata (backbone id, agent order, dims, scores) |
router_head.py |
Standalone reference implementation: embed a query, run the head, argmax |
Usage
pip install torch transformers numpy
python router_head.py "What is the derivative of x^3 + 2x?"
from router_head import pick_agent
result = pick_agent("What is the derivative of x^3 + 2x?")
print(result["picked_agent"]) # e.g. "frontier_a"
manifest.json's head_hidden_dim is 0 for this run, so the head is a pure linear layer: logits = embedding @ W + b, W shape (2048, 5), b shape (5,), unflattened from head_weights.npy in that order. The embedding step (mean-pool the backbone's last hidden state over real, non-padding tokens) must match exactly. See embed_query() in router_head.py.
Training data and held-out results
Trained on benchgen/router-pilot (reward matrix: 46 tasks x 5 agents x 3 repetitions, correctness measured, not assumed) joined with benchgen/router-pilot-tasks (prompt text) by task_id. Split: 35 train, 11 held-out test rows.
| Metric | Score |
|---|---|
| Train reward (CMA-ES fit) | 0.9429 |
| Held-out test reward | 0.8182 (27 of 33 graded calls) |
| Held-out random-agent baseline | 0.6364 |
| Held-out best-single-fixed-agent baseline | 0.7879 (26 of 33 graded calls) |
| Held-out per-question oracle (upper bound) | 0.8485 |
| Beats best fixed agent? | Yes, by one graded call out of 33. Not statistically meaningful |
The held-out set is 11 tasks with 3 attempts each. A difference of 0.03 is one graded call, so read this table as "no worse than the best single agent on this sample", not as proof that routing wins.
Benchmark on Benchgen
The same head is evaluated on the Router Fidelity Benchmark on Benchgen: 38 of 46 questions correct (82.6%), at roughly 3.1x lower cost than calling claude-sonnet-5 alone on the same 46 questions. These are single runs on an in-distribution set. The guide lists the per-slice results, including the slice where the single model won.
Run the same benchmark
- Open the Router Fidelity Benchmark (sign-in required to run).
- Register your head as a model, choose it on the Evaluate tab and run.
- Compare your pool and your head with ours. The guide has the exact steps.
Limitations
- In-distribution only. The reward data is English math, knowledge and reasoning tasks (MATH-500, MMLU, MMLU-Pro, ARC-Challenge, GSM8K, AIME2025). Outside that distribution (another language or a very different domain), the embeddings fall outside anything the head learned to separate, and its pick becomes closer to arbitrary than a real routing decision.
- Pool-specific. The head only knows how to choose between these 5 agents, in this order. A different pool needs retraining.
- Small and tie-heavy. Only 46 tasks carry reward labels (of 1,110 published prompts in
router-pilot-tasks), and 43 of them have ties between agents. Results are directional, not a large-sample benchmark. - Pilot verdict. The guide records that the pilot's dataset gate did not clear the routing signal. Treat this release as a worked example, not as a recommended production router.
License
Apache-2.0 for the head and the code. The backbone Qwen/Qwen3-1.7B is under its own license (Apache-2.0 on its model page). The training data is described in the two dataset cards, which list the license of every upstream source.
Citation
@misc{benchgen_router_lite_2026,
title = {Benchgen Router Lite: a linear router head on a frozen Qwen3-1.7B backbone},
author = {{Benchgen}},
year = {2026},
howpublished = {\url{https://huggingface.co/benchgen/benchgen-router-lite}},
note = {Trained on the Benchgen platform}
}
Provenance
Trained on the Benchgen platform. The idea of a small learned coordinator that picks among a pool of models was inspired by published work on LLM coordinators, including Sakana AI's Trinity. This release is Benchgen's own implementation, data and evaluation. We are not the authors of that work and are not affiliated with them.
