README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B
tags:
  - router
  - model-routing
  - cma-es
  - benchgen
datasets:
  - benchgen/router-pilot
  - benchgen/router-pilot-tasks
pipeline_tag: text-classification
---

# BenchGen Router Lite (`head-Qwen3-1.7B-e99b449fde`)

Repo: `benchgen/benchgen-router-lite`

A **router head**, not a fine-tuned LLM: a tiny (10,245-parameter) linear
classifier that sits on top of a frozen `Qwen/Qwen3-1.7B` backbone and learns
**which model in a pool should answer a given query**. The backbone never
updates — only this head does. Trained with separable CMA-ES (a
gradient-free evolutionary search) against a measured reward matrix, not
gradient descent, and never calls any pool model during training.

Full write-up (dataset, training run, benchmark results): [BenchGen Router
Lite docs guide](https://benchgen-docs-url/guides/router-head/benchgen-router-lite).

## How it works

```
query --> frozen Qwen3-1.7B (mean-pooled embedding, 2048-d)
      --> this head (linear: logits = embedding @ W + b)
      --> argmax over 5 pool agents
      --> picked agent's REAL reply is what gets returned
```

The router itself never generates an answer — it only picks who should.

## Pool (5 agents this head was trained to route between)

| Slot | Agent id (`agent_order`) | Underlying model | Role |
|---|---|---|---|
| Frontier | `frontier_a` | `openai/gpt-oss-120b` | Reasoning |
| Frontier | `frontier_b` | `deepseek/deepseek-v4-flash-0731` | Reasoning |
| Frontier | `frontier_c` | `google/gemma-3-27b-it` | Direct |
| Mid-tier | `open_mid` | `mistralai/mistral-nemo` | Direct (deliberately the weakest agent) |
| Cheap reasoning | `open_cheap_reasoning` | `inclusionai/ling-3.0-flash` | Reasoning |

The underlying model slugs above are **not baked into the head** — they're
the OpenRouter pool this specific run was trained against. A caller resolves
`picked_agent` to a real model separately (BenchGen's gateway does this via
its own pool config).

## Files

| File | What it is |
|---|---|
| `head_weights.npy` | Flat `float32` parameter vector (10,245 values) — `theta` |
| `manifest.json` | Architecture + training metadata (backbone id, agent order, dims, scores) |
| `router_head.py` | Standalone reference implementation: embed a query, run the head, argmax |

## Usage

```bash
pip install torch transformers numpy
python router_head.py "What is the derivative of x^3 + 2x?"
```

```python
from router_head import pick_agent

result = pick_agent("What is the derivative of x^3 + 2x?")
print(result["picked_agent"])   # e.g. "frontier_a"
```

`manifest.json`'s `head_hidden_dim` is `0` for this run, so the head is a
pure linear layer: `logits = embedding @ W + b`, `W` shape `(2048, 5)`, `b`
shape `(5,)`, unflattened from `head_weights.npy` in that order. The
embedding step (mean-pool the backbone's last hidden state over real,
non-padding tokens) **must match exactly** — see `router_head.py`'s
`embed_query()`.

## Training data & held-out results

Trained on [`benchgen/router-pilot`](https://huggingface.co/datasets/benchgen/router-pilot)
(reward matrix: 46 tasks x 5 agents x 3 repetitions, correctness measured, not
assumed) joined with [`benchgen/router-pilot-tasks`](https://huggingface.co/datasets/benchgen/router-pilot-tasks)
(prompt text) by `task_id`. Split 35 train / 11 held-out test rows.

| Metric | Score |
|---|---|
| Train reward (CMA-ES fit) | 0.9429 |
| **Held-out test reward** | **0.8182** |
| Held-out random-agent baseline | 0.6364 |
| Held-out best-single-fixed-agent baseline | 0.7879 |
| Held-out per-question oracle (upper bound) | 0.8485 |
| Beats best fixed agent? | **Yes** |

The router beats always-calling-the-best-single-agent on held-out data —
the actual test of whether the extra routing step is worth it over a naive
"always use one model" strategy.

## Limitations

- **In-distribution only.** The reward dataset is English math/knowledge/
  reasoning tasks (MATH500, MMLU, MMLU-Pro, ARC-Challenge, GSM8K, AIME2025).
  Evaluated outside that distribution (a different language, or a very
  different domain), the classifier's embeddings fall outside anything it
  learned to discriminate and its pick becomes closer to arbitrary than a
  real routing decision.
- **Pool-specific.** This head only knows how to choose between the exact
  5 agents listed above, in that exact order. Retraining is required for a
  different pool.
- Only 46 tasks currently carry reward labels (of 1,110 published prompts in
  `router-pilot-tasks`), so held-out numbers are directional, not a
  large-n benchmark result.

## Citation / provenance

Trained on the BenchGen platform. Architecture and training approach are
similar in spirit to Sakana AI's work on model routing, and the reward-matrix
pool design mirrors the coordinator setup in the Trinity paper
([arXiv:2512.04695](https://arxiv.org/abs/2512.04695)).