Benchgen: the benchmarking infrastructure for AI agents

Website: benchgen.com Run the benchmark on Benchgen Docs

GitHub X LinkedIn Hugging Face

Benchgen Router Lite (head-Qwen3-1.7B-e99b449fde)

Repo: benchgen/benchgen-router-lite · Built by Benchgen, the benchmarking infrastructure for AI agents.

A router head, not a fine-tuned LLM: a tiny (10,245-parameter) linear classifier that sits on top of a frozen Qwen/Qwen3-1.7B backbone and learns which model in a pool should answer a given query. The backbone never updates, only this head does. It is trained with separable CMA-ES (a gradient-free evolutionary search) against a measured reward matrix, not gradient descent, and it never calls any pool model during training.

Full write-up (dataset, training run, benchmark results): Router Lite guide.

Intended use

  • Research and teaching on LLM routing: how a small head can pick among a fixed pool of models.
  • A starting point you can retrain on your own pool and your own reward matrix.
  • A reproducible example of evaluating a router on Benchgen (see "Run the same benchmark" below).

Out of scope: production routing without retraining on your own pool and data, any language other than English, and domains outside math, knowledge and reasoning.

How it works

query --> frozen Qwen3-1.7B (mean-pooled embedding, 2048-d)
      --> this head (linear: logits = embedding @ W + b)
      --> argmax over 5 pool agents
      --> picked agent's REAL reply is what gets returned

The router itself never generates an answer. It only picks who should.

Pool (5 agents this head was trained to route between)

Slot Agent id (agent_order) Underlying model Role
Frontier frontier_a openai/gpt-oss-120b Reasoning
Frontier frontier_b deepseek/deepseek-v4-flash-0731 Reasoning
Frontier frontier_c google/gemma-3-27b-it Direct
Mid-tier open_mid mistralai/mistral-nemo Direct (deliberately the weakest agent)
Cheap reasoning open_cheap_reasoning inclusionai/ling-3.0-flash Reasoning

The underlying model slugs are not baked into the head. They are the OpenRouter pool this specific run was trained against. A caller resolves picked_agent to a real model separately (the Benchgen gateway does this through its own pool config).

Files

File What it is
head_weights.npy Flat float32 parameter vector (10,245 values), theta
manifest.json Architecture and training metadata (backbone id, agent order, dims, scores)
router_head.py Standalone reference implementation: embed a query, run the head, argmax

Usage

pip install torch transformers numpy
python router_head.py "What is the derivative of x^3 + 2x?"
from router_head import pick_agent

result = pick_agent("What is the derivative of x^3 + 2x?")
print(result["picked_agent"])   # e.g. "frontier_a"

manifest.json's head_hidden_dim is 0 for this run, so the head is a pure linear layer: logits = embedding @ W + b, W shape (2048, 5), b shape (5,), unflattened from head_weights.npy in that order. The embedding step (mean-pool the backbone's last hidden state over real, non-padding tokens) must match exactly. See embed_query() in router_head.py.

Training data and held-out results

Trained on benchgen/router-pilot (reward matrix: 46 tasks x 5 agents x 3 repetitions, correctness measured, not assumed) joined with benchgen/router-pilot-tasks (prompt text) by task_id. Split: 35 train, 11 held-out test rows.

Metric Score
Train reward (CMA-ES fit) 0.9429
Held-out test reward 0.8182 (27 of 33 graded calls)
Held-out random-agent baseline 0.6364
Held-out best-single-fixed-agent baseline 0.7879 (26 of 33 graded calls)
Held-out per-question oracle (upper bound) 0.8485
Beats best fixed agent? Yes, by one graded call out of 33. Not statistically meaningful

The held-out set is 11 tasks with 3 attempts each. A difference of 0.03 is one graded call, so read this table as "no worse than the best single agent on this sample", not as proof that routing wins.

Benchmark on Benchgen

The same head is evaluated on the Router Fidelity Benchmark on Benchgen: 38 of 46 questions correct (82.6%), at roughly 3.1x lower cost than calling claude-sonnet-5 alone on the same 46 questions. These are single runs on an in-distribution set. The guide lists the per-slice results, including the slice where the single model won.

Run the same benchmark

  1. Open the Router Fidelity Benchmark (sign-in required to run).
  2. Register your head as a model, choose it on the Evaluate tab and run.
  3. Compare your pool and your head with ours. The guide has the exact steps.

Limitations

  • In-distribution only. The reward data is English math, knowledge and reasoning tasks (MATH-500, MMLU, MMLU-Pro, ARC-Challenge, GSM8K, AIME2025). Outside that distribution (another language or a very different domain), the embeddings fall outside anything the head learned to separate, and its pick becomes closer to arbitrary than a real routing decision.
  • Pool-specific. The head only knows how to choose between these 5 agents, in this order. A different pool needs retraining.
  • Small and tie-heavy. Only 46 tasks carry reward labels (of 1,110 published prompts in router-pilot-tasks), and 43 of them have ties between agents. Results are directional, not a large-sample benchmark.
  • Pilot verdict. The guide records that the pilot's dataset gate did not clear the routing signal. Treat this release as a worked example, not as a recommended production router.

License

Apache-2.0 for the head and the code. The backbone Qwen/Qwen3-1.7B is under its own license (Apache-2.0 on its model page). The training data is described in the two dataset cards, which list the license of every upstream source.

Citation

@misc{benchgen_router_lite_2026,
  title        = {Benchgen Router Lite: a linear router head on a frozen Qwen3-1.7B backbone},
  author       = {{Benchgen}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/benchgen/benchgen-router-lite}},
  note         = {Trained on the Benchgen platform}
}

Provenance

Trained on the Benchgen platform. The idea of a small learned coordinator that picks among a pool of models was inspired by published work on LLM coordinators, including Sakana AI's Trinity. This release is Benchgen's own implementation, data and evaluation. We are not the authors of that work and are not affiliated with them.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for benchgen/benchgen-router-lite

Finetuned
Qwen/Qwen3-1.7B
Finetuned
(1,409)
this model

Datasets used to train benchgen/benchgen-router-lite