josefchen commited on
Commit
418bf00
·
verified ·
1 Parent(s): e38c605

Redesign Space and surface turnkey model evaluation

Browse files

Launch-system interface, copyable endpoint and checkpoint commands, dense reward demo, complete-run scorer, and unchanged verified release bundle.

Files changed (3) hide show
  1. README.md +49 -40
  2. app.py +629 -245
  3. assets/flavourbench-leaderboard.svg +0 -0
README.md CHANGED
@@ -1,8 +1,8 @@
1
  ---
2
  title: FlavourBench
3
  emoji: 🍲
4
- colorFrom: blue
5
- colorTo: blue
6
  sdk: gradio
7
  sdk_version: 6.9.0
8
  app_file: app.py
@@ -14,63 +14,72 @@ datasets:
14
 
15
  # FlavourBench
16
 
17
- **Josef Chen · Erim Hayretci**<br>
18
- Josef Chen, Independent Researcher · Erim Hayretci, Imperial College London
 
 
 
 
19
 
20
- This Space explores the final FlavourBench complete common core: 27 frontier endpoints, 534
21
- identical tasks per model, 14,418 valid scored responses, and all 351 paired comparisons.
 
 
 
 
22
 
23
- Epicure scores all 56 legal three-ingredient portfolios before a model is called. The Space lets
24
- you inspect the resulting leaderboard without relying on an LLM judge. It includes:
25
 
26
- - the full score table with simultaneous 95% intervals and bootstrap rank intervals;
27
- - statistical rank groups and all Holm-adjusted pairwise comparisons;
28
- - family profiles and score replication across two independently compiled panels; and
29
- - exact prompts, candidate lists, Epicure score maps, model answers, and response hashes.
30
 
31
- Every ranked endpoint has one valid response for every task. The Space bundle is content-addressed
32
- and makes no provider calls.
33
 
34
- ## Evaluate your own model
 
 
 
 
 
 
 
 
 
35
 
36
- The **Evaluate your model** tab scores a complete JSON or JSONL response artifact without receiving
37
- model credentials or weights. A public FlavourBench lab score is issued only when all 534 responses
38
- are present and parseable; incomplete runs retain coverage and per-task diagnostics. Uploads are not
39
- automatically added to the official leaderboard.
40
 
41
- Each JSONL row follows this compact contract:
42
 
43
  ```json
44
  {"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
45
  ```
46
 
47
- The Space also exposes named Gradio endpoints:
48
-
49
- - `/score_completion` performs one deterministic reward-map lookup;
50
- - `/score_submission` scores JSON or JSON Lines supplied as text;
51
- - `/training_reward` scores only the 426 non-leaderboard development maps; and
52
- - `/score_uploaded_submission` powers the file-upload interface.
53
 
54
- Use **Use via API** in the running Space for generated Python, JavaScript, and curl examples. The
55
- local SDK is preferable for high-throughput RL because it avoids network latency and Space rate
56
- limits. The Space exposes released FlavourBench reward maps, not the private Epicure corpus or a
57
- general arbitrary-task generation service.
58
 
59
- ## Train on separate reward maps
60
 
61
- The linked dataset now includes anchor-disjoint development splits for SFT, DPO, and GRPO. These
62
- training maps do not reuse any task ID or ingredient anchor from the 534-task official test set.
63
- Runnable LoRA recipes and the local dense reward function live in the source repository under
64
- `examples/lab`.
 
 
65
 
66
- Josef Chen is a Cohere Labs Catalyst Grant recipient. This acknowledgement does not imply Cohere
67
- endorsement of FlavourBench, Epicure, the protocol, or any model ranking.
 
68
 
69
- [Paper](https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf) |
70
- [Dataset](https://huggingface.co/datasets/josefchen/flavourbench) |
71
  [Source](https://github.com/josefchen/flavourbench)
72
 
73
- ## Citation
74
 
75
  ```bibtex
76
  @article{chen2026flavourbench,
 
1
  ---
2
  title: FlavourBench
3
  emoji: 🍲
4
+ colorFrom: red
5
+ colorTo: red
6
  sdk: gradio
7
  sdk_version: 6.9.0
8
  app_file: app.py
 
14
 
15
  # FlavourBench
16
 
17
+ ![Which AI knows food best? FlavourBench leaderboard](./assets/flavourbench-leaderboard.svg)
18
+
19
+ **Pick 3 ingredients from 8. Epicure scores all 56 legal portfolios first. Then every model faces
20
+ the same 534 decisions.**
21
+
22
+ The Space is both the public scorebook and a working benchmark interface:
23
 
24
+ - **Leaders** ranks all 27 endpoints with simultaneous intervals and statistical groups.
25
+ - **Profiles** breaks each score into substitution, pairing, and constraint performance.
26
+ - **Inspect** opens the exact prompt, answer, 56-choice score map, route, and content hashes.
27
+ - **Run your model** builds a copyable endpoint or local-checkpoint command, demonstrates one dense
28
+ training reward, and scores a complete JSONL run.
29
+ - **Compare** queries any of the 351 shared-task pairwise contrasts.
30
 
31
+ No model judge runs behind the interface. The Space performs deterministic lookups against the
32
+ released reward maps and makes no model-provider calls.
33
 
34
+ ## Run from your own environment
 
 
 
35
 
36
+ ```bash
37
+ python -m pip install "epicure-flavourbench @ git+https://github.com/josefchen/flavourbench.git"
38
 
39
+ export LAB_MODEL_API_KEY='...'
40
+ flavourbench run \
41
+ --backend openai-compatible \
42
+ --base-url https://your-endpoint.example/v1 \
43
+ --api-key-env LAB_MODEL_API_KEY \
44
+ --model your-exact-model-id \
45
+ --responses responses.jsonl \
46
+ --report flavourbench-report.json \
47
+ --resume
48
+ ```
49
 
50
+ Add `--limit 12` for a balanced smoke test. The runner checkpoints each answer and resumes without
51
+ repeating completed calls. Credentials and model weights stay in your environment.
 
 
52
 
53
+ The accepted response contract is one JSON object per line:
54
 
55
  ```json
56
  {"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
57
  ```
58
 
59
+ A comparable score requires one valid answer for all 534 tasks. Partial runs still receive
60
+ per-task and coverage diagnostics. Uploads are never added to the official leaderboard
61
+ automatically.
 
 
 
62
 
63
+ ## API and training
 
 
 
64
 
65
+ The Space exposes four named endpoints:
66
 
67
+ | Endpoint | Use |
68
+ |---|---|
69
+ | `/score_completion` | Score one completion on one official task |
70
+ | `/score_submission` | Score a complete JSON or JSONL artifact supplied as text |
71
+ | `/training_reward` | Query one of 426 anchor-disjoint development reward maps |
72
+ | `/score_uploaded_submission` | Score an uploaded artifact and return a report |
73
 
74
+ Use **Use via API** in the running Space for generated Python, JavaScript, and curl clients. For
75
+ high-throughput RL, use the local deterministic reward function. The linked dataset includes
76
+ ready-to-load SFT, DPO, and GRPO views plus runnable LoRA recipes for Hugging Face Jobs.
77
 
78
+ [Dataset and lab kit](https://huggingface.co/datasets/josefchen/flavourbench) ·
79
+ [Paper](https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf) ·
80
  [Source](https://github.com/josefchen/flavourbench)
81
 
82
+ Josef Chen, Independent Researcher · Erim Hayretci, Imperial College London
83
 
84
  ```bibtex
85
  @article{chen2026flavourbench,
app.py CHANGED
@@ -4,6 +4,7 @@ import hashlib
4
  import html
5
  import json
6
  import os
 
7
  import tempfile
8
  from datetime import UTC, datetime
9
  from pathlib import Path
@@ -25,249 +26,427 @@ BUNDLE_PATH = Path(
25
  )
26
  )
27
 
28
- BLUE = "#1769AA"
29
- CHARCOAL = "#262B33"
30
 
31
  CSS = """
32
  :root {
33
- --fb-accent: #1769AA;
34
- --fb-accent-soft: #EAF3FA;
35
- --fb-ink: #262B33;
36
- --fb-muted: #657180;
37
- --fb-paper: #F6F8FA;
38
- --fb-panel: #FFFFFF;
39
- --fb-rule: #D9E0E7;
40
- --fb-code: #EEF2F5;
41
  }
42
  .dark {
43
- --fb-accent-soft: #102B3E;
44
- --fb-ink: #EAF0F4;
45
- --fb-muted: #A7B2BC;
46
- --fb-paper: #11171D;
47
- --fb-panel: #182129;
48
- --fb-rule: #30404D;
49
- --fb-code: #202C35;
 
50
  }
 
51
  body, .gradio-container {
52
  background: var(--fb-paper) !important;
53
  color: var(--fb-ink) !important;
54
  font-family: "Geist", "Avenir Next", system-ui, sans-serif !important;
55
  }
56
- .gradio-container { max-width: 1460px !important; }
57
- .fb-shell { max-width: 1360px; margin: 0 auto; }
 
 
 
 
 
 
 
 
 
 
58
  .fb-hero {
59
  display: grid;
60
- grid-template-columns: minmax(0, .9fr) minmax(520px, 1.1fr);
61
- gap: 54px;
62
- padding: 48px 8px 34px;
63
- border-bottom: 1px solid var(--fb-rule);
64
  }
65
  .fb-kicker {
66
  color: var(--fb-accent);
67
  font-family: "IBM Plex Mono", ui-monospace, monospace;
68
- font-size: 12px;
69
- font-weight: 700;
70
- letter-spacing: .12em;
71
  text-transform: uppercase;
72
  }
73
  .fb-hero h1 {
74
  color: var(--fb-ink);
75
- font-size: clamp(46px, 6vw, 78px);
76
- letter-spacing: -.06em;
77
- line-height: .96;
78
- margin: 14px 0 18px;
79
- max-width: 760px;
 
80
  }
81
  .fb-dek {
82
  color: var(--fb-muted);
83
- font-size: 19px;
84
- line-height: 1.48;
85
  margin: 0;
86
- max-width: 640px;
87
- }
88
- .fb-byline {
89
- display: flex;
90
- flex-wrap: wrap;
91
- gap: 10px 22px;
92
- margin: 16px 0 0;
93
- }
94
- .fb-byline span {
95
- color: var(--fb-ink);
96
- display: flex;
97
- flex-direction: column;
98
- font-size: 14px;
99
- font-weight: 650;
100
- letter-spacing: .01em;
101
- }
102
- .fb-byline small {
103
- color: var(--fb-muted);
104
- font-size: 11px;
105
- font-weight: 500;
106
- letter-spacing: .04em;
107
- margin-top: 2px;
108
  }
109
  .fb-stats {
 
110
  display: grid;
111
- grid-template-columns: repeat(4, minmax(92px, 1fr));
112
- gap: 22px;
113
- margin-top: 31px;
 
114
  }
115
- .fb-stat { border-top: 2px solid var(--fb-rule); padding-top: 11px; }
116
  .fb-stat strong {
117
  color: var(--fb-ink);
118
  display: block;
119
- font-size: 26px;
 
120
  letter-spacing: -.04em;
121
  line-height: 1;
122
  }
123
  .fb-stat span {
124
  color: var(--fb-muted);
125
  display: block;
126
- font-size: 11px;
 
 
127
  margin-top: 7px;
128
  text-transform: uppercase;
129
- letter-spacing: .07em;
130
  }
131
  .fb-frontier {
132
  align-self: end;
133
- background: var(--fb-panel);
134
- border: 1px solid var(--fb-rule);
135
- border-top: 4px solid var(--fb-accent);
136
- border-radius: 8px;
137
- padding: 20px 22px 16px;
138
  }
139
  .fb-frontier-head {
140
  align-items: baseline;
141
  display: flex;
 
142
  justify-content: space-between;
143
- margin-bottom: 13px;
 
 
 
 
 
 
 
144
  }
145
- .fb-frontier-head strong { font-size: 14px; }
146
- .fb-frontier-head span { color: var(--fb-muted); font-size: 12px; }
147
  .fb-forest-row {
148
  align-items: center;
 
149
  display: grid;
150
- grid-template-columns: 150px 1fr 42px 38px;
151
- gap: 10px;
152
- min-height: 26px;
 
 
 
 
 
153
  }
154
  .fb-model {
155
  color: var(--fb-ink);
156
  font-size: 12px;
 
157
  overflow: hidden;
158
  text-overflow: ellipsis;
159
  white-space: nowrap;
160
  }
161
- .fb-axis { height: 13px; position: relative; }
 
 
 
162
  .fb-axis::before {
163
  background: var(--fb-rule);
164
  content: "";
165
- height: 1px;
166
  left: 0;
167
  position: absolute;
168
  right: 0;
169
- top: 6px;
170
- }
171
- .fb-ci {
172
- border-top: 2px solid var(--fb-muted);
173
- height: 1px;
174
- position: absolute;
175
- top: 5px;
176
  }
177
- .fb-ci::before, .fb-ci::after {
178
- background: var(--fb-muted);
179
- content: "";
180
- height: 7px;
181
  position: absolute;
182
- top: -4px;
183
- width: 1px;
184
  }
185
- .fb-ci::before { left: 0; }
186
- .fb-ci::after { right: 0; }
187
  .fb-point {
188
- background: var(--fb-accent);
189
- height: 11px;
 
190
  position: absolute;
191
- top: 1px;
192
- width: 3px;
 
193
  }
 
 
 
 
194
  .fb-number {
195
  color: var(--fb-ink);
196
  font-family: "IBM Plex Mono", ui-monospace, monospace;
197
- font-size: 11px;
 
198
  text-align: right;
199
  }
200
- .fb-group {
201
  color: var(--fb-muted);
 
202
  font-family: "IBM Plex Mono", ui-monospace, monospace;
203
- font-size: 10px;
204
- text-align: right;
 
 
 
 
 
205
  }
206
- .fb-section { margin: 27px 0 8px; }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
207
  .fb-section h2 {
208
  color: var(--fb-ink);
209
- font-size: 30px;
210
- letter-spacing: -.035em;
211
- margin: 0 0 4px;
 
 
 
 
 
 
 
 
 
212
  }
213
- .fb-section p { color: var(--fb-muted); margin: 0; max-width: 70ch; }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
214
  .fb-metric-grid {
 
 
215
  display: grid;
216
  grid-template-columns: repeat(3, 1fr);
217
- gap: 12px;
218
- margin: 8px 0 16px;
219
  }
220
- .fb-metric {
221
- background: var(--fb-panel);
222
- border: 1px solid var(--fb-rule);
223
- border-radius: 8px;
224
- padding: 16px;
225
  }
 
226
  .fb-metric small {
227
  color: var(--fb-muted);
228
  display: block;
229
- font-size: 11px;
 
230
  letter-spacing: .05em;
231
  text-transform: uppercase;
232
  }
233
- .fb-metric strong { color: var(--fb-ink); display: block; font-size: 23px; margin-top: 5px; }
 
 
 
 
 
 
 
234
  .fb-evidence {
235
- background: var(--fb-accent-soft);
236
- border-left: 4px solid var(--fb-accent);
237
- border-radius: 0 8px 8px 0;
238
  color: var(--fb-ink);
239
- line-height: 1.48;
240
- padding: 17px 19px;
241
  }
 
242
  .fb-evidence code, .fb-hash {
243
  background: var(--fb-code);
244
  color: var(--fb-muted);
245
  font-family: "IBM Plex Mono", ui-monospace, monospace;
246
  font-size: 11px;
 
247
  }
248
  .fb-method {
249
  display: grid;
250
- grid-template-columns: 1.15fr .85fr;
251
- gap: 28px;
 
 
 
 
 
 
 
 
 
 
 
 
252
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
253
  .fb-footer {
254
  border-top: 1px solid var(--fb-rule);
255
  color: var(--fb-muted);
 
256
  font-size: 12px;
257
- margin-top: 36px;
258
- padding: 18px 8px 26px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
259
  }
260
- .gradio-dataframe, .block { border-radius: 8px !important; }
261
- @media (max-width: 980px) {
262
- .fb-hero, .fb-method { grid-template-columns: 1fr; }
263
- .fb-hero { gap: 28px; padding-top: 34px; }
264
  }
265
- @media (max-width: 700px) {
266
- .fb-hero h1 { font-size: 48px; }
267
- .fb-stats, .fb-metric-grid { grid-template-columns: repeat(2, 1fr); }
268
- .fb-forest-row { grid-template-columns: 105px 1fr 36px 30px; }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
269
  }
270
  @media (prefers-reduced-motion: reduce) {
 
271
  *, *::before, *::after { animation: none !important; transition: none !important; }
272
  }
273
  """
@@ -341,6 +520,58 @@ TASK_LABEL_TO_ID = {
341
  f"{row['task_id']} | {str(row['family']).replace('_', ' ')}": str(row["task_id"])
342
  for row in TASKS
343
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
344
 
345
 
346
  def _completion_diagnostic(model_id: str) -> dict[str, Any]:
@@ -378,61 +609,51 @@ def _completion_diagnostic(model_id: str) -> dict[str, Any]:
378
  }
379
 
380
 
381
- def _short(value: str) -> str:
382
- return (
383
- value.replace("GPT-5.6 ", "5.6 ")
384
- .replace("Claude ", "")
385
- .replace("DeepSeek ", "DS ")
386
- .replace("Command ", "Cmd ")
387
- )
388
-
389
-
390
  def _frontier_html() -> str:
 
 
391
  rows = []
392
- for model in DISPLAY_MODELS[:10]:
393
  score = float(model["flavourbench_score"])
394
- lower, upper = (float(value) for value in model["score_simultaneous_95_ci"])
395
- left = max(0.0, min(100.0, lower))
396
- right = max(left, min(100.0, upper))
397
- group = model.get("statistical_rank_group")
398
  rows.append(
399
  "<div class='fb-forest-row'>"
400
- f"<div class='fb-model' title='{html.escape(str(model['model_name']))}'>"
401
- f"{html.escape(_short(str(model['model_name'])))}</div>"
 
402
  "<div class='fb-axis'>"
403
- f"<span class='fb-ci' style='left:{left:.3f}%;width:{right - left:.3f}%'></span>"
404
- f"<span class='fb-point' style='left:{score:.3f}%'></span>"
405
  "</div>"
406
  f"<div class='fb-number'>{score:.1f}</div>"
407
- f"<div class='fb-group'>G{group if group is not None else '-'}</div>"
408
  "</div>"
409
  )
410
- return "".join(rows)
 
 
 
411
 
412
 
413
  def _hero_html() -> str:
414
  inference = BUNDLE["analysis"]["inference"]
415
- panel_phrase = f" across {PANEL_COUNT} collection panels" if PANEL_COUNT > 1 else ""
416
  return f"""
417
  <div class="fb-shell fb-hero">
418
  <section>
419
- <div class="fb-kicker">Executable culinary evaluation</div>
420
- <h1>{TASK_COUNT:,} decisions.<br>No model judge.</h1>
421
- <p class="fb-dek">Executable score maps rank {MODEL_COUNT} frontier endpoints{panel_phrase}
422
- with anchor-clustered uncertainty and inspectable responses.</p>
423
- <div class="fb-byline">
424
- <span>Josef Chen<small>Independent Researcher</small></span>
425
- <span>Erim Hayretci<small>Imperial College London</small></span>
426
- </div>
427
  <div class="fb-stats">
428
- <div class="fb-stat"><strong>{MODEL_COUNT}</strong><span>models</span></div>
429
- <div class="fb-stat"><strong>{TASK_COUNT}</strong><span>tasks</span></div>
430
- <div class="fb-stat"><strong>{PRIMARY_COUNT:,}</strong><span>primary cells</span></div>
431
- <div class="fb-stat"><strong>{inference["pairwise_hypotheses"]}</strong><span>paired tests</span></div>
432
  </div>
433
  </section>
434
- <section class="fb-frontier" aria-label="Score forest with simultaneous intervals">
435
- <div class="fb-frontier-head"><strong>FlavourBench Score</strong><span>Top 10, simultaneous 95%</span></div>
436
  {_frontier_html()}
437
  </section>
438
  </div>
@@ -447,18 +668,48 @@ def _leaderboard_frame() -> pd.DataFrame:
447
  rows.append(
448
  {
449
  "Rank": model["point_estimate_rank"],
450
- "Group": model["statistical_rank_group"],
451
  "Model": model["model_name"],
452
- "Score": round(float(model["flavourbench_score"]), 2),
453
  "Simultaneous 95%": f"{ci[0]:.2f} to {ci[1]:.2f}",
 
454
  "Rank 95%": f"{rank_ci[0]} to {rank_ci[1]}",
455
  "Cells": f"{model['coverage']['valid_scored']}/{TASK_COUNT}",
456
- "Backend": model["execution_backend"],
457
  }
458
  )
459
  return pd.DataFrame(rows)
460
 
461
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
462
  def _model_detail(model_name: str) -> tuple[str, pd.DataFrame]:
463
  model = MODEL_BY_NAME[model_name]
464
  rank_interval = model["bootstrap_rank_95_interval"]
@@ -614,6 +865,55 @@ def _training_reward_api(task_id: str, completion: str) -> dict[str, Any]:
614
  }
615
 
616
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
617
  def _score_upload(
618
  artifact_path: str | None,
619
  model_name: str,
@@ -689,62 +989,56 @@ def _score_upload(
689
 
690
  theme = gr.themes.Base(
691
  primary_hue=gr.themes.Color(
692
- c50="#EAF3FA",
693
- c100="#D5E8F5",
694
- c200="#A9D0E9",
695
- c300="#75B3DA",
696
- c400="#4292C6",
697
- c500=BLUE,
698
- c600="#12588F",
699
- c700="#104873",
700
- c800="#103C5D",
701
- c900="#10334E",
702
- c950="#081E30",
703
  ),
704
- neutral_hue="slate",
705
  font=gr.themes.GoogleFont("Geist", weights=(400, 500, 600, 700)),
706
  font_mono=gr.themes.GoogleFont("IBM Plex Mono", weights=(400, 600)),
707
  ).set(
708
- body_background_fill="#F6F8FA",
709
- block_background_fill="#FFFFFF",
710
- block_border_width="1px",
711
  block_label_text_weight="600",
712
- button_primary_background_fill=BLUE,
713
- button_primary_background_fill_hover="#12588F",
714
  )
715
 
716
 
717
- with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
718
  gr.HTML(_hero_html())
719
  with gr.Tabs():
720
- with gr.Tab("Leaderboard"):
721
  gr.HTML(
722
- """
723
  <div class="fb-section">
724
- <h2>The complete common-core leaderboard</h2>
725
- <p>Every model is scored on the same 534 tasks. Point ranks sit beside statistical groups and simultaneous intervals.</p>
726
  </div>
727
  """
728
  )
729
- gr.Dataframe(
730
- value=_leaderboard_frame(),
731
- interactive=False,
732
- wrap=True,
733
- show_search="filter",
734
- show_row_numbers=False,
735
- column_widths=[55, 55, 245, 80, 180, 110, 125, 120],
736
- )
737
  gr.Markdown(
738
- "A point rank orders the observed scores. A statistical group keeps models together "
739
- "when the shared-task evidence does not separate them after multiplicity control."
 
740
  )
741
 
742
- with gr.Tab("Model profile"):
743
  gr.HTML(
744
  """
745
  <div class="fb-section">
746
- <h2>Family profile and panel replication</h2>
747
- <p>Inspect where a model earns its score and how its estimate moves across the two independently compiled panels.</p>
748
  </div>
749
  """
750
  )
@@ -765,14 +1059,15 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
765
  _model_detail,
766
  inputs=model_selector,
767
  outputs=[model_summary, family_table],
 
768
  )
769
 
770
- with gr.Tab("Task lens"):
771
  gr.HTML(
772
  """
773
  <div class="fb-section">
774
  <h2>Open one scored decision</h2>
775
- <p>Read the prompt, the model answer, and the top of the precomputed reward surface.</p>
776
  </div>
777
  """
778
  )
@@ -791,36 +1086,120 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
791
  filterable=True,
792
  scale=2,
793
  )
794
- inspect_task = gr.Button("Inspect", variant="primary", scale=0)
795
  initial = _task_detail(MODEL_NAMES[0], next(iter(TASK_LABEL_TO_ID)))
796
  task_status = gr.HTML(initial[0])
797
- prompt = gr.Textbox(value=initial[1], label="Exact prompt", lines=13, interactive=False)
798
- choices = gr.JSON(value=initial[2], label="Candidates")
799
  score_map = gr.Dataframe(
800
  value=initial[3],
801
- label="Top 12 of 56 frozen selections",
802
  interactive=False,
803
  wrap=True,
804
  show_row_numbers=False,
805
  )
806
  answer = gr.Markdown(value=initial[4], label="Model response")
807
- provenance = gr.Markdown(value=initial[5], label="Provenance")
808
  inspect_task.click(
809
  _task_detail,
810
  inputs=[task_model, task_selector],
811
  outputs=[task_status, prompt, choices, score_map, answer, provenance],
 
812
  )
813
 
814
- with gr.Tab("Evaluate your model"):
815
  gr.HTML(
816
  f"""
817
  <div class="fb-section">
818
- <h2>Bring a checkpoint or endpoint</h2>
819
- <p>Run the prompts in your own environment, upload the response artifact, and score it here. Credentials and model weights never enter this Space.</p>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
820
  </div>
821
- <div class="fb-evidence">
822
- <strong>Comparable means complete.</strong> A lab score requires one valid answer for every one of the {TASK_COUNT} released tasks. Partial runs receive coverage and per-task diagnostics only.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
823
  </div>
 
824
  """
825
  )
826
  with gr.Row():
@@ -845,7 +1224,7 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
845
  file_types=[".jsonl", ".json"],
846
  type="filepath",
847
  )
848
- score_upload = gr.Button("Score artifact", variant="primary")
849
  lab_summary = gr.Markdown()
850
  lab_rows = gr.Dataframe(
851
  interactive=False,
@@ -863,26 +1242,22 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
863
  )
864
  gr.Markdown(
865
  """
866
- The accepted JSONL contract is one object per task:
867
-
868
  ```json
869
  {"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
870
  ```
871
 
872
- For automation, use the named `score_completion`, `score_submission`, and `training_reward`
873
- endpoints shown under **Use via API** in the Space footer. `training_reward` accepts only the 426
874
- development task IDs; it cannot score against the leaderboard by accident. The source repository
875
- also provides a local runner, scorer, schemas, and TRL recipes. Local reward lookup remains the
876
- recommended path for high-throughput training.
877
  """
878
  )
879
 
880
- with gr.Tab("Pairwise evidence"):
881
  gr.HTML(
882
  f"""
883
  <div class="fb-section">
884
- <h2>Is the score gap resolved?</h2>
885
- <p>Query any paired contrast from the {PAIR_COUNT}-test family.</p>
886
  </div>
887
  """
888
  )
@@ -899,34 +1274,45 @@ recommended path for high-throughput training.
899
  label="Second model",
900
  filterable=True,
901
  )
902
- compare = gr.Button("Compare", variant="primary", scale=0)
903
  pair_result = gr.HTML(_pair_detail(MODEL_NAMES[0], MODEL_NAMES[1]))
904
- compare.click(_pair_detail, inputs=[left_model, right_model], outputs=pair_result)
 
 
 
 
 
905
 
906
- with gr.Tab("Method and download"):
907
  gr.HTML(
908
  f"""
909
  <div class="fb-section">
910
- <h2>One metric, complete evidence</h2>
911
- <p>The Space makes no model or provider calls. Its lab API performs deterministic lookups against the released reward maps.</p>
 
 
 
 
 
912
  </div>
913
  <div class="fb-method">
914
  <div>
915
  <h3>Scoring contract</h3>
916
- <p>Every task exposes eight candidates and all 56 three-item scores. The
917
- FlavourBench Score is the equal-family mean across substitution, pairing, and
918
- constraint tasks. The release uses a complete 27-by-534 matrix: every ranked
919
- model contributes one valid response to every scored task.</p>
920
  <h3>Inference</h3>
921
  <p>Results use {INDEPENDENT_CLUSTER_COUNT:,} ingredient-anchor clusters,
922
- 50,000 ingredient-anchor cluster bootstraps, simultaneous score bands,
923
- 100,000 cluster sign flips, Holm correction, exact-chance tests, bootstrap rank
924
- intervals, and an independently compiled second panel.</p>
 
 
 
925
  </div>
926
  <aside class="fb-evidence">
927
- <strong>Exact release</strong><br>
928
  <span class="fb-hash">{BUNDLE["release_artifact_sha256"]}</span><br><br>
929
- {MODEL_COUNT} models<br>{TASK_COUNT} tasks<br>{INDEPENDENT_CLUSTER_COUNT:,} anchor clusters<br>{PRIMARY_COUNT:,} complete scored responses<br>{BUNDLE["analysis"]["resolved_pair_count"]}/{PAIR_COUNT} resolved pairs
930
  </aside>
931
  </div>
932
  """
@@ -936,12 +1322,9 @@ recommended path for high-throughput training.
936
  ```bash
937
  git clone https://github.com/josefchen/flavourbench.git
938
  cd flavourbench
939
- pip install -e '.[dev]'
940
- make -C paper -f Makefile.powered verify
941
- make -C paper -f Makefile.powered arxiv
942
  ```
943
-
944
- [Paper](https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf) | [Dataset](https://huggingface.co/datasets/josefchen/flavourbench) | [Source](https://github.com/josefchen/flavourbench)
945
  """
946
  )
947
 
@@ -967,7 +1350,8 @@ make -C paper -f Makefile.powered arxiv
967
  gr.HTML(
968
  """
969
  <div class="fb-shell fb-footer">
970
- FlavourBench | Josef Chen, Independent Researcher · Erim Hayretci, Imperial College London
 
971
  </div>
972
  """
973
  )
 
4
  import html
5
  import json
6
  import os
7
+ import shlex
8
  import tempfile
9
  from datetime import UTC, datetime
10
  from pathlib import Path
 
26
  )
27
  )
28
 
29
+ RUST = "#B54135"
30
+ INK = "#171815"
31
 
32
  CSS = """
33
  :root {
34
+ --fb-accent: #B54135;
35
+ --fb-accent-soft: #F1DDD8;
36
+ --fb-ink: #171815;
37
+ --fb-muted: #666960;
38
+ --fb-paper: #F4F3EE;
39
+ --fb-paper-raised: #FAF9F5;
40
+ --fb-rule: #C9CBC4;
41
+ --fb-code: #E8E7E1;
42
  }
43
  .dark {
44
+ --fb-accent: #EF796D;
45
+ --fb-accent-soft: #422723;
46
+ --fb-ink: #F0EFE9;
47
+ --fb-muted: #A9ACA3;
48
+ --fb-paper: #171815;
49
+ --fb-paper-raised: #20211E;
50
+ --fb-rule: #41433D;
51
+ --fb-code: #2A2B27;
52
  }
53
+ html { scroll-behavior: smooth; }
54
  body, .gradio-container {
55
  background: var(--fb-paper) !important;
56
  color: var(--fb-ink) !important;
57
  font-family: "Geist", "Avenir Next", system-ui, sans-serif !important;
58
  }
59
+ .gradio-container {
60
+ max-width: none !important;
61
+ padding: 0 !important;
62
+ }
63
+ .gradio-container > main { padding: 0 !important; }
64
+ .fb-shell {
65
+ box-sizing: border-box;
66
+ margin: 0 auto;
67
+ max-width: 1440px;
68
+ padding-left: clamp(22px, 4vw, 62px);
69
+ padding-right: clamp(22px, 4vw, 62px);
70
+ }
71
  .fb-hero {
72
  display: grid;
73
+ gap: clamp(40px, 6vw, 96px);
74
+ grid-template-columns: minmax(360px, .82fr) minmax(540px, 1.18fr);
75
+ padding-bottom: 42px;
76
+ padding-top: clamp(46px, 6vw, 86px);
77
  }
78
  .fb-kicker {
79
  color: var(--fb-accent);
80
  font-family: "IBM Plex Mono", ui-monospace, monospace;
81
+ font-size: 11px;
82
+ font-weight: 600;
83
+ letter-spacing: .13em;
84
  text-transform: uppercase;
85
  }
86
  .fb-hero h1 {
87
  color: var(--fb-ink);
88
+ font-size: clamp(56px, 6vw, 92px);
89
+ font-weight: 650;
90
+ letter-spacing: -.065em;
91
+ line-height: .91;
92
+ margin: 18px 0 23px;
93
+ max-width: 740px;
94
  }
95
  .fb-dek {
96
  color: var(--fb-muted);
97
+ font-size: clamp(17px, 1.6vw, 21px);
98
+ line-height: 1.46;
99
  margin: 0;
100
+ max-width: 570px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
101
  }
102
  .fb-stats {
103
+ border-top: 1px solid var(--fb-ink);
104
  display: grid;
105
+ gap: 20px;
106
+ grid-template-columns: repeat(4, minmax(88px, 1fr));
107
+ margin-top: 36px;
108
+ padding-top: 13px;
109
  }
 
110
  .fb-stat strong {
111
  color: var(--fb-ink);
112
  display: block;
113
+ font-size: 25px;
114
+ font-weight: 620;
115
  letter-spacing: -.04em;
116
  line-height: 1;
117
  }
118
  .fb-stat span {
119
  color: var(--fb-muted);
120
  display: block;
121
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
122
+ font-size: 10px;
123
+ letter-spacing: .06em;
124
  margin-top: 7px;
125
  text-transform: uppercase;
 
126
  }
127
  .fb-frontier {
128
  align-self: end;
129
+ border-top: 1px solid var(--fb-ink);
130
+ min-width: 0;
131
+ padding-top: 12px;
 
 
132
  }
133
  .fb-frontier-head {
134
  align-items: baseline;
135
  display: flex;
136
+ gap: 16px;
137
  justify-content: space-between;
138
+ margin-bottom: 15px;
139
+ }
140
+ .fb-frontier-head strong { font-size: 15px; font-weight: 620; }
141
+ .fb-frontier-head span {
142
+ color: var(--fb-muted);
143
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
144
+ font-size: 10px;
145
+ text-align: right;
146
  }
 
 
147
  .fb-forest-row {
148
  align-items: center;
149
+ border-bottom: 1px solid var(--fb-rule);
150
  display: grid;
151
+ gap: 12px;
152
+ grid-template-columns: 22px minmax(130px, 190px) 1fr 47px;
153
+ min-height: 36px;
154
+ }
155
+ .fb-place {
156
+ color: var(--fb-muted);
157
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
158
+ font-size: 10px;
159
  }
160
  .fb-model {
161
  color: var(--fb-ink);
162
  font-size: 12px;
163
+ font-weight: 560;
164
  overflow: hidden;
165
  text-overflow: ellipsis;
166
  white-space: nowrap;
167
  }
168
+ .fb-axis {
169
+ height: 10px;
170
+ position: relative;
171
+ }
172
  .fb-axis::before {
173
  background: var(--fb-rule);
174
  content: "";
175
+ height: 2px;
176
  left: 0;
177
  position: absolute;
178
  right: 0;
179
+ top: 4px;
 
 
 
 
 
 
180
  }
181
+ .fb-bar {
182
+ background: #A7ABA4;
183
+ height: 4px;
184
+ left: 0;
185
  position: absolute;
186
+ top: 3px;
 
187
  }
 
 
188
  .fb-point {
189
+ background: var(--fb-ink);
190
+ border-radius: 50%;
191
+ height: 10px;
192
  position: absolute;
193
+ top: 0;
194
+ transform: translateX(-50%);
195
+ width: 10px;
196
  }
197
+ .fb-forest-row:first-of-type .fb-bar,
198
+ .fb-forest-row:first-of-type .fb-point { background: var(--fb-accent); }
199
+ .fb-forest-row:first-of-type .fb-model,
200
+ .fb-forest-row:first-of-type .fb-number { color: var(--fb-accent); }
201
  .fb-number {
202
  color: var(--fb-ink);
203
  font-family: "IBM Plex Mono", ui-monospace, monospace;
204
+ font-size: 12px;
205
+ font-weight: 600;
206
  text-align: right;
207
  }
208
+ .fb-chart-foot {
209
  color: var(--fb-muted);
210
+ display: flex;
211
  font-family: "IBM Plex Mono", ui-monospace, monospace;
212
+ font-size: 9px;
213
+ justify-content: space-between;
214
+ margin: 9px 59px 0 224px;
215
+ }
216
+ .tab-wrapper {
217
+ border-bottom: 1px solid var(--fb-rule);
218
+ border-top: 1px solid var(--fb-rule);
219
  }
220
+ .tab-nav {
221
+ margin: 0 auto !important;
222
+ max-width: 1316px !important;
223
+ padding-left: clamp(22px, 4vw, 62px) !important;
224
+ padding-right: clamp(22px, 4vw, 62px) !important;
225
+ }
226
+ .tab-nav button {
227
+ border: 0 !important;
228
+ color: var(--fb-muted) !important;
229
+ font-size: 13px !important;
230
+ padding: 15px 0 13px !important;
231
+ margin-right: 30px !important;
232
+ }
233
+ .tab-nav button.selected {
234
+ border-bottom: 2px solid var(--fb-accent) !important;
235
+ color: var(--fb-ink) !important;
236
+ }
237
+ .overflow-menu { display: none !important; }
238
+ .tab-container.visually-hidden { display: none !important; }
239
+ .tabitem {
240
+ box-sizing: border-box;
241
+ margin: 0 auto !important;
242
+ max-width: 1440px !important;
243
+ padding: 0 clamp(22px, 4vw, 62px) !important;
244
+ }
245
+ .fb-section { margin: 36px 0 18px; }
246
  .fb-section h2 {
247
  color: var(--fb-ink);
248
+ font-size: clamp(30px, 3vw, 45px);
249
+ font-weight: 620;
250
+ letter-spacing: -.045em;
251
+ line-height: 1.02;
252
+ margin: 0 0 9px;
253
+ }
254
+ .fb-section p {
255
+ color: var(--fb-muted);
256
+ font-size: 15px;
257
+ line-height: 1.5;
258
+ margin: 0;
259
+ max-width: 72ch;
260
  }
261
+ .fb-step-line {
262
+ border-bottom: 1px solid var(--fb-rule);
263
+ border-top: 1px solid var(--fb-rule);
264
+ display: grid;
265
+ grid-template-columns: repeat(3, 1fr);
266
+ margin: 8px 0 26px;
267
+ }
268
+ .fb-step { padding: 18px 22px 20px 0; }
269
+ .fb-step + .fb-step { border-left: 1px solid var(--fb-rule); padding-left: 22px; }
270
+ .fb-step b {
271
+ color: var(--fb-accent);
272
+ display: block;
273
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
274
+ font-size: 10px;
275
+ letter-spacing: .08em;
276
+ margin-bottom: 7px;
277
+ text-transform: uppercase;
278
+ }
279
+ .fb-step strong { display: block; font-size: 17px; margin-bottom: 4px; }
280
+ .fb-step span { color: var(--fb-muted); font-size: 13px; line-height: 1.45; }
281
  .fb-metric-grid {
282
+ border-bottom: 1px solid var(--fb-rule);
283
+ border-top: 1px solid var(--fb-rule);
284
  display: grid;
285
  grid-template-columns: repeat(3, 1fr);
286
+ margin: 10px 0 22px;
 
287
  }
288
+ .fb-metric { min-width: 0; padding: 19px 22px 21px 0; }
289
+ .fb-metric:nth-child(3n+2), .fb-metric:nth-child(3n+3) {
290
+ border-left: 1px solid var(--fb-rule);
291
+ padding-left: 22px;
 
292
  }
293
+ .fb-metric:nth-child(n+4) { border-top: 1px solid var(--fb-rule); }
294
  .fb-metric small {
295
  color: var(--fb-muted);
296
  display: block;
297
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
298
+ font-size: 10px;
299
  letter-spacing: .05em;
300
  text-transform: uppercase;
301
  }
302
+ .fb-metric strong {
303
+ color: var(--fb-ink);
304
+ display: block;
305
+ font-size: 24px;
306
+ font-weight: 620;
307
+ margin-top: 5px;
308
+ overflow-wrap: anywhere;
309
+ }
310
  .fb-evidence {
311
+ border-bottom: 1px solid var(--fb-rule);
312
+ border-top: 1px solid var(--fb-rule);
 
313
  color: var(--fb-ink);
314
+ line-height: 1.52;
315
+ padding: 15px 0;
316
  }
317
+ .fb-evidence strong:first-child { color: var(--fb-accent); }
318
  .fb-evidence code, .fb-hash {
319
  background: var(--fb-code);
320
  color: var(--fb-muted);
321
  font-family: "IBM Plex Mono", ui-monospace, monospace;
322
  font-size: 11px;
323
+ overflow-wrap: anywhere;
324
  }
325
  .fb-method {
326
  display: grid;
327
+ gap: clamp(30px, 6vw, 86px);
328
+ grid-template-columns: 1.25fr .75fr;
329
+ }
330
+ .fb-method h3 { font-size: 19px; margin: 23px 0 6px; }
331
+ .fb-method p { color: var(--fb-muted); line-height: 1.55; }
332
+ .fb-command-note {
333
+ color: var(--fb-muted);
334
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
335
+ font-size: 10px;
336
+ line-height: 1.5;
337
+ }
338
+ .fb-table-wrap {
339
+ overflow-x: auto;
340
+ width: 100%;
341
  }
342
+ .fb-table {
343
+ border: 0 !important;
344
+ color: var(--fb-ink);
345
+ font-family: "IBM Plex Mono", ui-monospace, monospace;
346
+ font-size: 12px;
347
+ min-width: 860px;
348
+ width: 100%;
349
+ }
350
+ .fb-table caption {
351
+ height: 1px;
352
+ overflow: hidden;
353
+ position: absolute;
354
+ width: 1px;
355
+ }
356
+ .fb-table th {
357
+ border-left: 0 !important;
358
+ border-right: 0 !important;
359
+ border-top: 0 !important;
360
+ border-bottom: 1px solid var(--fb-ink);
361
+ color: var(--fb-muted);
362
+ font-size: 10px;
363
+ font-weight: 600;
364
+ letter-spacing: .05em;
365
+ padding: 10px 12px 11px 0;
366
+ text-align: left;
367
+ text-transform: uppercase;
368
+ }
369
+ .fb-table td {
370
+ border-left: 0 !important;
371
+ border-right: 0 !important;
372
+ border-top: 0 !important;
373
+ border-bottom: 1px solid var(--fb-rule);
374
+ padding: 11px 12px 11px 0;
375
+ vertical-align: top;
376
+ }
377
+ .fb-table .fb-score-cell { font-weight: 600; }
378
+ .fb-table tbody tr:first-child .fb-rank-cell,
379
+ .fb-table tbody tr:first-child .fb-score-cell { color: var(--fb-accent); }
380
+ .fb-table tbody tr:hover { background: color-mix(in srgb, var(--fb-ink) 3%, transparent); }
381
  .fb-footer {
382
  border-top: 1px solid var(--fb-rule);
383
  color: var(--fb-muted);
384
+ display: flex;
385
  font-size: 12px;
386
+ justify-content: space-between;
387
+ margin-top: 54px;
388
+ padding-bottom: 30px;
389
+ padding-top: 18px;
390
+ }
391
+ .fb-footer a { color: var(--fb-ink); text-decoration: none; }
392
+ .fb-footer a:hover { color: var(--fb-accent); text-decoration: underline; }
393
+ .gradio-container footer a, .gradio-container footer button { color: var(--fb-muted) !important; }
394
+ .block, .form, .gradio-dataframe, .gradio-json, .gradio-textbox {
395
+ border-radius: 0 !important;
396
+ box-shadow: none !important;
397
+ }
398
+ .block:not(.gradio-dataframe):not(.gradio-json):not(.gradio-textbox) {
399
+ border-color: transparent !important;
400
+ }
401
+ button.primary {
402
+ background: var(--fb-ink) !important;
403
+ border: 1px solid var(--fb-ink) !important;
404
+ border-radius: 0 !important;
405
+ color: var(--fb-paper) !important;
406
+ min-height: 44px !important;
407
+ }
408
+ button.primary:hover { background: var(--fb-accent) !important; border-color: var(--fb-accent) !important; }
409
+ input, textarea, select {
410
+ border-radius: 0 !important;
411
+ box-shadow: none !important;
412
  }
413
+ input:focus, textarea:focus, select:focus, button:focus-visible, a:focus-visible {
414
+ outline: 2px solid var(--fb-accent) !important;
415
+ outline-offset: 2px !important;
 
416
  }
417
+ table { border-collapse: collapse !important; }
418
+ th { background: var(--fb-paper) !important; }
419
+ pre, code { border-radius: 0 !important; }
420
+ @media (max-width: 1040px) {
421
+ .fb-hero { grid-template-columns: 1fr; }
422
+ .fb-frontier { max-width: 820px; }
423
+ .fb-method { grid-template-columns: 1fr; }
424
+ }
425
+ @media (max-width: 720px) {
426
+ .fb-hero { gap: 38px; padding-bottom: 32px; }
427
+ .fb-hero h1 { font-size: clamp(52px, 15vw, 72px); }
428
+ .fb-stats { gap: 16px 10px; grid-template-columns: repeat(2, 1fr); }
429
+ .fb-forest-row { gap: 8px; grid-template-columns: 19px minmax(96px, 142px) 1fr 42px; }
430
+ .fb-model { font-size: 11px; }
431
+ .fb-chart-foot { margin-left: 133px; margin-right: 50px; }
432
+ .fb-chart-foot span:nth-child(2) { display: none; }
433
+ .fb-step-line { grid-template-columns: 1fr; }
434
+ .fb-step + .fb-step { border-left: 0; border-top: 1px solid var(--fb-rule); padding-left: 0; }
435
+ .fb-metric-grid { grid-template-columns: repeat(2, 1fr); }
436
+ .fb-metric:nth-child(3n+2), .fb-metric:nth-child(3n+3) { border-left: 0; padding-left: 0; }
437
+ .fb-metric:nth-child(even) { border-left: 1px solid var(--fb-rule); padding-left: 18px; }
438
+ .fb-metric:nth-child(n+3) { border-top: 1px solid var(--fb-rule); }
439
+ .tab-nav { overflow-x: auto !important; }
440
+ .tab-nav button { margin-right: 22px !important; white-space: nowrap !important; }
441
+ .fb-footer { display: block; }
442
+ .fb-footer span { display: block; margin-top: 7px; }
443
+ }
444
+ @media (max-width: 430px) {
445
+ .fb-forest-row { grid-template-columns: 18px minmax(82px, 114px) 1fr 39px; }
446
+ .fb-chart-foot { margin-left: 109px; margin-right: 47px; }
447
  }
448
  @media (prefers-reduced-motion: reduce) {
449
+ html { scroll-behavior: auto; }
450
  *, *::before, *::after { animation: none !important; transition: none !important; }
451
  }
452
  """
 
520
  f"{row['task_id']} | {str(row['family']).replace('_', ' ')}": str(row["task_id"])
521
  for row in TASKS
522
  }
523
+ LAB_TASK_LABEL_TO_ID = {
524
+ f"{row['task_id']} | {str(row['family']).replace('_', ' ')} | {row['lab_split']}": str(
525
+ row["task_id"]
526
+ )
527
+ for row in LAB_TASKS
528
+ }
529
+
530
+
531
+ def _lab_name(row: dict[str, Any]) -> str:
532
+ model_name = str(row["model_name"])
533
+ prefixes = (
534
+ ("SpaceXAI:", "xAI"),
535
+ ("Anthropic:", "Anthropic"),
536
+ ("Claude ", "Anthropic"),
537
+ ("MoonshotAI:", "Kimi"),
538
+ ("ByteDance Seed:", "ByteDance"),
539
+ ("Thinking Machines:", "Thinking Machines"),
540
+ ("MiniMax:", "MiniMax"),
541
+ ("DeepSeek:", "DeepSeek"),
542
+ ("OpenAI:", "OpenAI"),
543
+ ("Google:", "Google"),
544
+ ("Meta:", "Meta"),
545
+ ("Qwen:", "Qwen"),
546
+ ("Tencent:", "Tencent"),
547
+ ("Z.ai:", "Z.ai"),
548
+ ("NVIDIA:", "NVIDIA"),
549
+ ("Cohere:", "Cohere"),
550
+ ("Mistral:", "Mistral"),
551
+ )
552
+ for prefix, lab in prefixes:
553
+ if model_name.startswith(prefix):
554
+ return lab
555
+ return str(row.get("provider_name") or model_name.split(":", 1)[0])
556
+
557
+
558
+ def _model_label(model_name: str) -> str:
559
+ label = model_name.split(":", 1)[-1].strip()
560
+ return (
561
+ label.replace("GPT-5.6 ", "5.6 ")
562
+ .replace("Claude ", "")
563
+ .replace("DeepSeek ", "")
564
+ .replace("Command ", "")
565
+ )
566
+
567
+
568
+ _seen_labs: set[str] = set()
569
+ LAB_CHAMPIONS: list[dict[str, Any]] = []
570
+ for _model in DISPLAY_MODELS:
571
+ _lab = _lab_name(_model)
572
+ if _lab not in _seen_labs:
573
+ _seen_labs.add(_lab)
574
+ LAB_CHAMPIONS.append(_model)
575
 
576
 
577
  def _completion_diagnostic(model_id: str) -> dict[str, Any]:
 
609
  }
610
 
611
 
 
 
 
 
 
 
 
 
 
612
  def _frontier_html() -> str:
613
+ axis_floor = 55.0
614
+ axis_ceiling = 66.0
615
  rows = []
616
+ for place, model in enumerate(LAB_CHAMPIONS[:10], start=1):
617
  score = float(model["flavourbench_score"])
618
+ position = max(0.0, min(100.0, (score - axis_floor) / (axis_ceiling - axis_floor) * 100))
619
+ full_name = str(model["model_name"])
620
+ lab = _lab_name(model)
621
+ label = _model_label(full_name)
622
  rows.append(
623
  "<div class='fb-forest-row'>"
624
+ f"<div class='fb-place'>{place:02d}</div>"
625
+ f"<div class='fb-model' title='{html.escape(full_name)}'>"
626
+ f"{html.escape(lab)} · {html.escape(label)}</div>"
627
  "<div class='fb-axis'>"
628
+ f"<span class='fb-bar' style='width:{position:.3f}%'></span>"
629
+ f"<span class='fb-point' style='left:{position:.3f}%'></span>"
630
  "</div>"
631
  f"<div class='fb-number'>{score:.1f}</div>"
 
632
  "</div>"
633
  )
634
+ return "".join(rows) + (
635
+ f"<div class='fb-chart-foot'><span>{axis_floor:.0f}</span>"
636
+ f"<span>FlavourBench Score</span><span>{axis_ceiling:.0f}</span></div>"
637
+ )
638
 
639
 
640
  def _hero_html() -> str:
641
  inference = BUNDLE["analysis"]["inference"]
 
642
  return f"""
643
  <div class="fb-shell fb-hero">
644
  <section>
645
+ <div class="fb-kicker">FlavourBench / executable benchmark</div>
646
+ <h1>Which AI knows food best?</h1>
647
+ <p class="fb-dek">Epicure scores every legal answer first. Then {MODEL_COUNT} frontier endpoints face the same {TASK_COUNT} food decisions.</p>
 
 
 
 
 
648
  <div class="fb-stats">
649
+ <div class="fb-stat"><strong>{MODEL_COUNT}</strong><span>frontier endpoints</span></div>
650
+ <div class="fb-stat"><strong>{TASK_COUNT}</strong><span>shared tasks</span></div>
651
+ <div class="fb-stat"><strong>{PRIMARY_COUNT:,}</strong><span>scored answers</span></div>
652
+ <div class="fb-stat"><strong>{inference["pairwise_hypotheses"]}</strong><span>model pairs</span></div>
653
  </div>
654
  </section>
655
+ <section class="fb-frontier" aria-label="Best FlavourBench model from each leading lab">
656
+ <div class="fb-frontier-head"><strong>Best model from each lab</strong><span>Point estimates · focused 55 to 66 axis</span></div>
657
  {_frontier_html()}
658
  </section>
659
  </div>
 
668
  rows.append(
669
  {
670
  "Rank": model["point_estimate_rank"],
 
671
  "Model": model["model_name"],
672
+ "Score": round(float(model["flavourbench_score"]), 2),
673
  "Simultaneous 95%": f"{ci[0]:.2f} to {ci[1]:.2f}",
674
+ "Group": model["statistical_rank_group"],
675
  "Rank 95%": f"{rank_ci[0]} to {rank_ci[1]}",
676
  "Cells": f"{model['coverage']['valid_scored']}/{TASK_COUNT}",
 
677
  }
678
  )
679
  return pd.DataFrame(rows)
680
 
681
 
682
+ def _leaderboard_html() -> str:
683
+ rows = []
684
+ for model in DISPLAY_MODELS:
685
+ ci = model["score_simultaneous_95_ci"]
686
+ rank_ci = model["bootstrap_rank_95_interval"]
687
+ rows.append(
688
+ "<tr>"
689
+ f"<td class='fb-rank-cell'>{int(model['point_estimate_rank']):02d}</td>"
690
+ f"<td>{html.escape(str(model['model_name']))}</td>"
691
+ f"<td class='fb-score-cell'>{float(model['flavourbench_score']):.2f}</td>"
692
+ f"<td>{float(ci[0]):.2f} to {float(ci[1]):.2f}</td>"
693
+ f"<td>G{model['statistical_rank_group']}</td>"
694
+ f"<td>{rank_ci[0]} to {rank_ci[1]}</td>"
695
+ f"<td>{model['coverage']['valid_scored']}/{TASK_COUNT}</td>"
696
+ "</tr>"
697
+ )
698
+ return """
699
+ <div class="fb-table-wrap">
700
+ <table class="fb-table">
701
+ <caption>Complete FlavourBench common-core leaderboard</caption>
702
+ <thead><tr>
703
+ <th scope="col">Rank</th><th scope="col">Model</th><th scope="col">Score</th>
704
+ <th scope="col">Simultaneous 95%</th><th scope="col">Group</th>
705
+ <th scope="col">Rank 95%</th><th scope="col">Cells</th>
706
+ </tr></thead>
707
+ <tbody>""" + "".join(rows) + """</tbody>
708
+ </table>
709
+ </div>
710
+ """
711
+
712
+
713
  def _model_detail(model_name: str) -> tuple[str, pd.DataFrame]:
714
  model = MODEL_BY_NAME[model_name]
715
  rank_interval = model["bootstrap_rank_95_interval"]
 
865
  }
866
 
867
 
868
+ def _command_preview(
869
+ runtime: str,
870
+ model: str,
871
+ base_url: str,
872
+ api_key_env: str,
873
+ scope: str,
874
+ ) -> str:
875
+ """Render a copyable lab command without receiving model credentials."""
876
+
877
+ model = " ".join(str(model or "").split()) or "your-exact-model-id"
878
+ base_url = " ".join(str(base_url or "").split()) or "https://your-endpoint.example/v1"
879
+ api_key_env = "".join(character for character in str(api_key_env or "") if character.isalnum() or character == "_")
880
+ api_key_env = api_key_env or "LAB_MODEL_API_KEY"
881
+ smoke = " --limit 12" if scope.startswith("12-task") else ""
882
+ backend = "transformers" if runtime.startswith("Local") else "openai-compatible"
883
+ route = ""
884
+ if backend == "openai-compatible":
885
+ route = f" \\\n+ --base-url {shlex.quote(base_url)} \\\n+ --api-key-env {shlex.quote(api_key_env)}"
886
+ command = (
887
+ "python -m pip install \"epicure-flavourbench @ "
888
+ "git+https://github.com/josefchen/flavourbench.git\"\n\n"
889
+ f"flavourbench run \\\n+ --backend {backend} \\\n+ --model {shlex.quote(model)}"
890
+ f"{route} \\\n+ --responses responses.jsonl \\\n+ --report flavourbench-report.json \\\n+ --resume{smoke}"
891
+ )
892
+ note = (
893
+ f"Set `{api_key_env}` in your shell before running. The key stays on your machine."
894
+ if backend == "openai-compatible"
895
+ else "The checkpoint runs locally through Transformers. No endpoint credential is used."
896
+ )
897
+ return f"```bash\n{command}\n```\n\n{note}"
898
+
899
+
900
+ def _reward_preview(task_label: str, completion: str) -> str:
901
+ """Score one development completion for the interactive reward demonstration."""
902
+
903
+ task_id = LAB_TASK_LABEL_TO_ID[task_label]
904
+ try:
905
+ result = _training_reward_api(task_id, completion)
906
+ except SpaceLabError as error:
907
+ return f"<div class='fb-evidence'><strong>Not parseable.</strong> {html.escape(str(error))}</div>"
908
+ selection = html.escape(str(result.get("selection") or result.get("observed_selection") or ""))
909
+ return f"""
910
+ <div class="fb-evidence">
911
+ <strong>Reward {float(result['reward']):.4f}</strong> · score {float(result['score']):.2f} · selection <code>{selection}</code><br>
912
+ This map belongs to the anchor-disjoint development track. It cannot alter the public leaderboard.
913
+ </div>
914
+ """
915
+
916
+
917
  def _score_upload(
918
  artifact_path: str | None,
919
  model_name: str,
 
989
 
990
  theme = gr.themes.Base(
991
  primary_hue=gr.themes.Color(
992
+ c50="#F8ECE9",
993
+ c100="#F1DDD8",
994
+ c200="#E7BCB5",
995
+ c300="#D98F84",
996
+ c400="#C96357",
997
+ c500=RUST,
998
+ c600="#98352D",
999
+ c700="#7C2E28",
1000
+ c800="#662A25",
1001
+ c900="#562722",
1002
+ c950="#2F120F",
1003
  ),
1004
+ neutral_hue="stone",
1005
  font=gr.themes.GoogleFont("Geist", weights=(400, 500, 600, 700)),
1006
  font_mono=gr.themes.GoogleFont("IBM Plex Mono", weights=(400, 600)),
1007
  ).set(
1008
+ body_background_fill="#F4F3EE",
1009
+ block_background_fill="#F4F3EE",
1010
+ block_border_width="0px",
1011
  block_label_text_weight="600",
1012
+ button_primary_background_fill=INK,
1013
+ button_primary_background_fill_hover=RUST,
1014
  )
1015
 
1016
 
1017
+ with gr.Blocks(title="FlavourBench | Which AI knows food best?") as demo:
1018
  gr.HTML(_hero_html())
1019
  with gr.Tabs():
1020
+ with gr.Tab("Leaders"):
1021
  gr.HTML(
1022
+ f"""
1023
  <div class="fb-section">
1024
+ <h2>The full scoreboard</h2>
1025
+ <p>One score for the same {TASK_COUNT} tasks. Intervals and rank groups show which gaps the shared-task evidence can actually resolve.</p>
1026
  </div>
1027
  """
1028
  )
1029
+ gr.HTML(_leaderboard_html())
 
 
 
 
 
 
 
1030
  gr.Markdown(
1031
+ "**Read the score first.** The focused chart above uses point estimates. "
1032
+ "The table adds simultaneous 95% bands, bootstrap rank intervals, and multiplicity-controlled groups. "
1033
+ "A score of 100 means selecting Epicure's optimum on every task."
1034
  )
1035
 
1036
+ with gr.Tab("Profiles"):
1037
  gr.HTML(
1038
  """
1039
  <div class="fb-section">
1040
+ <h2>Where each model wins</h2>
1041
+ <p>Break the headline score into substitution, pairing, and constraint performance, then compare the two collection panels.</p>
1042
  </div>
1043
  """
1044
  )
 
1059
  _model_detail,
1060
  inputs=model_selector,
1061
  outputs=[model_summary, family_table],
1062
+ api_visibility="private",
1063
  )
1064
 
1065
+ with gr.Tab("Inspect"):
1066
  gr.HTML(
1067
  """
1068
  <div class="fb-section">
1069
  <h2>Open one scored decision</h2>
1070
+ <p>Every answer is traceable to the exact prompt, model response, and precomputed 56-choice reward surface.</p>
1071
  </div>
1072
  """
1073
  )
 
1086
  filterable=True,
1087
  scale=2,
1088
  )
1089
+ inspect_task = gr.Button("Inspect task", variant="primary", scale=0)
1090
  initial = _task_detail(MODEL_NAMES[0], next(iter(TASK_LABEL_TO_ID)))
1091
  task_status = gr.HTML(initial[0])
1092
+ prompt = gr.Textbox(value=initial[1], label="Exact prompt", lines=11, interactive=False)
1093
+ choices = gr.JSON(value=initial[2], label="Eight candidates")
1094
  score_map = gr.Dataframe(
1095
  value=initial[3],
1096
+ label="Top 12 of 56 scored selections",
1097
  interactive=False,
1098
  wrap=True,
1099
  show_row_numbers=False,
1100
  )
1101
  answer = gr.Markdown(value=initial[4], label="Model response")
1102
+ provenance = gr.Markdown(value=initial[5], label="Content hashes and route")
1103
  inspect_task.click(
1104
  _task_detail,
1105
  inputs=[task_model, task_selector],
1106
  outputs=[task_status, prompt, choices, score_map, answer, provenance],
1107
+ api_visibility="private",
1108
  )
1109
 
1110
+ with gr.Tab("Run your model"):
1111
  gr.HTML(
1112
  f"""
1113
  <div class="fb-section">
1114
+ <h2>Run FlavourBench on any model</h2>
1115
+ <p>Your endpoint key or checkpoint stays in your environment. The open runner fetches the exact task set, resumes interrupted jobs, and writes a verifiable report.</p>
1116
+ </div>
1117
+ <div class="fb-step-line">
1118
+ <div class="fb-step"><b>01 / Connect</b><strong>Endpoint or checkpoint</strong><span>OpenAI-compatible APIs and local Transformers models use the same task contract.</span></div>
1119
+ <div class="fb-step"><b>02 / Run</b><strong>{TASK_COUNT} shared decisions</strong><span>Start with 12 tasks, then resume into the complete run without repeating finished calls.</span></div>
1120
+ <div class="fb-step"><b>03 / Verify</b><strong>Score plus uncertainty</strong><span>The report binds the task set, responses, score, coverage, and inference by hash.</span></div>
1121
+ </div>
1122
+ """
1123
+ )
1124
+ with gr.Row():
1125
+ with gr.Column(scale=1):
1126
+ runtime = gr.Radio(
1127
+ choices=["Hosted OpenAI-compatible endpoint", "Local Transformers checkpoint"],
1128
+ value="Hosted OpenAI-compatible endpoint",
1129
+ label="Runtime",
1130
+ )
1131
+ run_model = gr.Textbox(
1132
+ value="your-exact-model-id",
1133
+ label="Model ID or checkpoint",
1134
+ )
1135
+ run_scope = gr.Radio(
1136
+ choices=["12-task smoke test", f"Full {TASK_COUNT}-task benchmark"],
1137
+ value="12-task smoke test",
1138
+ label="Run size",
1139
+ )
1140
+ with gr.Column(scale=1):
1141
+ run_base_url = gr.Textbox(
1142
+ value="https://your-endpoint.example/v1",
1143
+ label="Base URL for hosted endpoints",
1144
+ )
1145
+ run_key_env = gr.Textbox(
1146
+ value="LAB_MODEL_API_KEY",
1147
+ label="Local environment variable containing the key",
1148
+ )
1149
+ generate_command = gr.Button("Build command", variant="primary")
1150
+ initial_command = _command_preview(
1151
+ "Hosted OpenAI-compatible endpoint",
1152
+ "your-exact-model-id",
1153
+ "https://your-endpoint.example/v1",
1154
+ "LAB_MODEL_API_KEY",
1155
+ "12-task smoke test",
1156
+ )
1157
+ run_command = gr.Markdown(initial_command)
1158
+ generate_command.click(
1159
+ _command_preview,
1160
+ inputs=[runtime, run_model, run_base_url, run_key_env, run_scope],
1161
+ outputs=run_command,
1162
+ api_visibility="private",
1163
+ )
1164
+
1165
+ gr.HTML(
1166
+ """
1167
+ <div class="fb-section">
1168
+ <h2>Try the training reward</h2>
1169
+ <p>Paste one completion and query an anchor-disjoint development map. This is the same deterministic reward used by the local GRPO recipe.</p>
1170
  </div>
1171
+ """
1172
+ )
1173
+ initial_lab_label = next(iter(LAB_TASK_LABEL_TO_ID))
1174
+ with gr.Row():
1175
+ reward_task = gr.Dropdown(
1176
+ choices=list(LAB_TASK_LABEL_TO_ID),
1177
+ value=initial_lab_label,
1178
+ label="Development task",
1179
+ filterable=True,
1180
+ scale=2,
1181
+ )
1182
+ reward_completion = gr.Textbox(
1183
+ value="FINAL_SELECTION: A,B,C",
1184
+ label="Model completion",
1185
+ scale=2,
1186
+ )
1187
+ score_reward = gr.Button("Score answer", variant="primary", scale=0)
1188
+ reward_result = gr.HTML(_reward_preview(initial_lab_label, "FINAL_SELECTION: A,B,C"))
1189
+ score_reward.click(
1190
+ _reward_preview,
1191
+ inputs=[reward_task, reward_completion],
1192
+ outputs=reward_result,
1193
+ api_visibility="private",
1194
+ )
1195
+
1196
+ gr.HTML(
1197
+ f"""
1198
+ <div class="fb-section">
1199
+ <h2>Score a completed run</h2>
1200
+ <p>Upload one JSON or JSONL response per task. Complete runs receive a FlavourBench Score; partial runs receive diagnostics only.</p>
1201
  </div>
1202
+ <div class="fb-evidence"><strong>Comparable means complete.</strong> All {TASK_COUNT} responses must be present and parseable. Uploads are scored in-session and never added to the public leaderboard automatically.</div>
1203
  """
1204
  )
1205
  with gr.Row():
 
1224
  file_types=[".jsonl", ".json"],
1225
  type="filepath",
1226
  )
1227
+ score_upload = gr.Button("Score complete artifact", variant="primary")
1228
  lab_summary = gr.Markdown()
1229
  lab_rows = gr.Dataframe(
1230
  interactive=False,
 
1242
  )
1243
  gr.Markdown(
1244
  """
 
 
1245
  ```json
1246
  {"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
1247
  ```
1248
 
1249
+ The named `score_completion`, `score_submission`, and `training_reward` endpoints appear under
1250
+ **Use via API**. For high-throughput RL, use the local reward lookup and the runnable SFT, DPO,
1251
+ and GRPO recipes in the source repository.
 
 
1252
  """
1253
  )
1254
 
1255
+ with gr.Tab("Compare"):
1256
  gr.HTML(
1257
  f"""
1258
  <div class="fb-section">
1259
+ <h2>Does the gap hold up?</h2>
1260
+ <p>Query any of the {PAIR_COUNT} paired model contrasts on the same tasks, with Holm control across the full comparison family.</p>
1261
  </div>
1262
  """
1263
  )
 
1274
  label="Second model",
1275
  filterable=True,
1276
  )
1277
+ compare = gr.Button("Compare models", variant="primary", scale=0)
1278
  pair_result = gr.HTML(_pair_detail(MODEL_NAMES[0], MODEL_NAMES[1]))
1279
+ compare.click(
1280
+ _pair_detail,
1281
+ inputs=[left_model, right_model],
1282
+ outputs=pair_result,
1283
+ api_visibility="private",
1284
+ )
1285
 
1286
+ with gr.Tab("Method"):
1287
  gr.HTML(
1288
  f"""
1289
  <div class="fb-section">
1290
+ <h2>One lookup, repeated {TASK_COUNT} times</h2>
1291
+ <p>The Space makes no provider calls. It reads released reward maps and returns deterministic scores.</p>
1292
+ </div>
1293
+ <div class="fb-step-line">
1294
+ <div class="fb-step"><b>Task</b><strong>Pick 3 of 8</strong><span>Each prompt defines exactly 56 legal portfolios.</span></div>
1295
+ <div class="fb-step"><b>Ground truth</b><strong>Score all 56 first</strong><span>Epicure constructs the task map before any evaluated model runs.</span></div>
1296
+ <div class="fb-step"><b>Benchmark</b><strong>Aggregate 534 choices</strong><span>The metric equal-weights substitution, pairing, and constraint families.</span></div>
1297
  </div>
1298
  <div class="fb-method">
1299
  <div>
1300
  <h3>Scoring contract</h3>
1301
+ <p>A task score ranges from 0 to 100 on its released Epicure map. The complete
1302
+ release is a {MODEL_COUNT} by {TASK_COUNT} matrix with one valid response in every cell.</p>
 
 
1303
  <h3>Inference</h3>
1304
  <p>Results use {INDEPENDENT_CLUSTER_COUNT:,} ingredient-anchor clusters,
1305
+ 50,000 shared cluster bootstraps, simultaneous score bands, 100,000 cluster
1306
+ sign flips, Holm correction, exact tests against a random legal choice, bootstrap rank intervals,
1307
+ and an independently compiled second panel.</p>
1308
+ <h3>Training boundary</h3>
1309
+ <p>The 426 SFT, DPO, and GRPO development maps use anchors that do not occur in
1310
+ the {TASK_COUNT}-task leaderboard. Training cannot query the official test reward maps by accident.</p>
1311
  </div>
1312
  <aside class="fb-evidence">
1313
+ <strong>Content-addressed release</strong><br>
1314
  <span class="fb-hash">{BUNDLE["release_artifact_sha256"]}</span><br><br>
1315
+ {MODEL_COUNT} endpoints<br>{TASK_COUNT} tasks<br>{INDEPENDENT_CLUSTER_COUNT:,} anchor clusters<br>{PRIMARY_COUNT:,} complete answers<br>{BUNDLE["analysis"]["resolved_pair_count"]}/{PAIR_COUNT} resolved pairs
1316
  </aside>
1317
  </div>
1318
  """
 
1322
  ```bash
1323
  git clone https://github.com/josefchen/flavourbench.git
1324
  cd flavourbench
1325
+ python -m pip install -e '.[dev]'
1326
+ pytest -q tests/lab_cli_test.py tests/hf_lab_space_api_test.py
 
1327
  ```
 
 
1328
  """
1329
  )
1330
 
 
1350
  gr.HTML(
1351
  """
1352
  <div class="fb-shell fb-footer">
1353
+ <div>FlavourBench · Josef Chen, Independent Researcher · Erim Hayretci, Imperial College London</div>
1354
+ <span><a href="https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf">Paper</a> · <a href="https://huggingface.co/datasets/josefchen/flavourbench">Dataset</a> · <a href="https://github.com/josefchen/flavourbench">Source</a></span>
1355
  </div>
1356
  """
1357
  )
assets/flavourbench-leaderboard.svg ADDED