kfallah commited on
Commit
15216e2
·
verified ·
1 Parent(s): 27192f3

Benchmarks: shipped-artifact live per-source results + savings column

Browse files
Files changed (1) hide show
  1. README.md +10 -11
README.md CHANGED
@@ -35,22 +35,21 @@ At inference: embed the request, compare it against a bank of tasks with measure
35
  for every model, and pick the cheapest model predicted to solve it. If nothing in the bank
36
  is similar to the request, the router does not guess — it escalates to the strongest model.
37
 
38
- <!-- Benchmarks are frozen per Kion (2026-08-03): the DeepSWE 6-seed results below are the
39
- numbers of record for this card. Do not replace or restate them. -->
40
 
41
  ## Benchmarks
42
 
43
- Measured on DeepSWE (real long-horizon software engineering tasks), 6 holdout seeds —
44
- the router never saw the held-out tasks or their repositories during fitting:
45
 
46
- | policy | quality | $ / task |
47
- |---|---|---|
48
- | **this router** | **0.948** | **$2.70** |
49
- | always opus-5 (high) — strongest single model | 0.964 | $6.20 |
50
- | best single model picked without hindsight (gpt-5.5 / opus-5, varies by seed) | 0.934 | $5.70 |
51
 
52
- The router matches the strongest single model's quality within noise at **2.2× lower
53
- cost**, and beats committing to any single model chosen up front on both quality and cost.
54
 
55
  ## Scope
56
 
 
35
  for every model, and pick the cheapest model predicted to solve it. If nothing in the bank
36
  is similar to the request, the router does not guess — it escalates to the strongest model.
37
 
38
+ <!-- Benchmarks section format set by Kion (2026-08-03): the shipped artifact's live-measured
39
+ per-source results, with a savings column. Plain model names, no protocol jargon. -->
40
 
41
  ## Benchmarks
42
 
43
+ Live-measured on held-out tasks the router never saw during fitting, against always using
44
+ opus-5 (high), the strongest single model:
45
 
46
+ | task type | router quality | opus-5 quality | cheaper | saved |
47
+ |---|---|---|---|---|
48
+ | short interactive requests | 1.000 | 1.000 | **6.0×** | **83%** |
49
+ | competitive programming (LiveCodeBench) | 0.961 | 0.997 | **6.0×** | **83%** |
50
+ | long-horizon software tasks (DeepSWE) | 0.908 | 0.926 | **3.8×** | **74%** |
51
 
52
+ Quality differences are within statistical noise on every task type; the saving is the win.
 
53
 
54
  ## Scope
55