Spaces:
Running
Running
Redesign Space and surface turnkey model evaluation
Browse filesLaunch-system interface, copyable endpoint and checkpoint commands, dense reward demo, complete-run scorer, and unchanged verified release bundle.
- README.md +49 -40
- app.py +629 -245
- assets/flavourbench-leaderboard.svg +0 -0
README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
---
|
| 2 |
title: FlavourBench
|
| 3 |
emoji: 🍲
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.9.0
|
| 8 |
app_file: app.py
|
|
@@ -14,63 +14,72 @@ datasets:
|
|
| 14 |
|
| 15 |
# FlavourBench
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
-
|
| 24 |
-
|
| 25 |
|
| 26 |
-
|
| 27 |
-
- statistical rank groups and all Holm-adjusted pairwise comparisons;
|
| 28 |
-
- family profiles and score replication across two independently compiled panels; and
|
| 29 |
-
- exact prompts, candidate lists, Epicure score maps, model answers, and response hashes.
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
are present and parseable; incomplete runs retain coverage and per-task diagnostics. Uploads are not
|
| 39 |
-
automatically added to the official leaderboard.
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
```json
|
| 44 |
{"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
|
| 45 |
```
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
- `/score_submission` scores JSON or JSON Lines supplied as text;
|
| 51 |
-
- `/training_reward` scores only the 426 non-leaderboard development maps; and
|
| 52 |
-
- `/score_uploaded_submission` powers the file-upload interface.
|
| 53 |
|
| 54 |
-
|
| 55 |
-
local SDK is preferable for high-throughput RL because it avoids network latency and Space rate
|
| 56 |
-
limits. The Space exposes released FlavourBench reward maps, not the private Epicure corpus or a
|
| 57 |
-
general arbitrary-task generation service.
|
| 58 |
|
| 59 |
-
|
| 60 |
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
`
|
|
|
|
|
|
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
|
|
|
| 68 |
|
| 69 |
-
[
|
| 70 |
-
[
|
| 71 |
[Source](https://github.com/josefchen/flavourbench)
|
| 72 |
|
| 73 |
-
|
| 74 |
|
| 75 |
```bibtex
|
| 76 |
@article{chen2026flavourbench,
|
|
|
|
| 1 |
---
|
| 2 |
title: FlavourBench
|
| 3 |
emoji: 🍲
|
| 4 |
+
colorFrom: red
|
| 5 |
+
colorTo: red
|
| 6 |
sdk: gradio
|
| 7 |
sdk_version: 6.9.0
|
| 8 |
app_file: app.py
|
|
|
|
| 14 |
|
| 15 |
# FlavourBench
|
| 16 |
|
| 17 |
+

|
| 18 |
+
|
| 19 |
+
**Pick 3 ingredients from 8. Epicure scores all 56 legal portfolios first. Then every model faces
|
| 20 |
+
the same 534 decisions.**
|
| 21 |
+
|
| 22 |
+
The Space is both the public scorebook and a working benchmark interface:
|
| 23 |
|
| 24 |
+
- **Leaders** ranks all 27 endpoints with simultaneous intervals and statistical groups.
|
| 25 |
+
- **Profiles** breaks each score into substitution, pairing, and constraint performance.
|
| 26 |
+
- **Inspect** opens the exact prompt, answer, 56-choice score map, route, and content hashes.
|
| 27 |
+
- **Run your model** builds a copyable endpoint or local-checkpoint command, demonstrates one dense
|
| 28 |
+
training reward, and scores a complete JSONL run.
|
| 29 |
+
- **Compare** queries any of the 351 shared-task pairwise contrasts.
|
| 30 |
|
| 31 |
+
No model judge runs behind the interface. The Space performs deterministic lookups against the
|
| 32 |
+
released reward maps and makes no model-provider calls.
|
| 33 |
|
| 34 |
+
## Run from your own environment
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
+
```bash
|
| 37 |
+
python -m pip install "epicure-flavourbench @ git+https://github.com/josefchen/flavourbench.git"
|
| 38 |
|
| 39 |
+
export LAB_MODEL_API_KEY='...'
|
| 40 |
+
flavourbench run \
|
| 41 |
+
--backend openai-compatible \
|
| 42 |
+
--base-url https://your-endpoint.example/v1 \
|
| 43 |
+
--api-key-env LAB_MODEL_API_KEY \
|
| 44 |
+
--model your-exact-model-id \
|
| 45 |
+
--responses responses.jsonl \
|
| 46 |
+
--report flavourbench-report.json \
|
| 47 |
+
--resume
|
| 48 |
+
```
|
| 49 |
|
| 50 |
+
Add `--limit 12` for a balanced smoke test. The runner checkpoints each answer and resumes without
|
| 51 |
+
repeating completed calls. Credentials and model weights stay in your environment.
|
|
|
|
|
|
|
| 52 |
|
| 53 |
+
The accepted response contract is one JSON object per line:
|
| 54 |
|
| 55 |
```json
|
| 56 |
{"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
|
| 57 |
```
|
| 58 |
|
| 59 |
+
A comparable score requires one valid answer for all 534 tasks. Partial runs still receive
|
| 60 |
+
per-task and coverage diagnostics. Uploads are never added to the official leaderboard
|
| 61 |
+
automatically.
|
|
|
|
|
|
|
|
|
|
| 62 |
|
| 63 |
+
## API and training
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
+
The Space exposes four named endpoints:
|
| 66 |
|
| 67 |
+
| Endpoint | Use |
|
| 68 |
+
|---|---|
|
| 69 |
+
| `/score_completion` | Score one completion on one official task |
|
| 70 |
+
| `/score_submission` | Score a complete JSON or JSONL artifact supplied as text |
|
| 71 |
+
| `/training_reward` | Query one of 426 anchor-disjoint development reward maps |
|
| 72 |
+
| `/score_uploaded_submission` | Score an uploaded artifact and return a report |
|
| 73 |
|
| 74 |
+
Use **Use via API** in the running Space for generated Python, JavaScript, and curl clients. For
|
| 75 |
+
high-throughput RL, use the local deterministic reward function. The linked dataset includes
|
| 76 |
+
ready-to-load SFT, DPO, and GRPO views plus runnable LoRA recipes for Hugging Face Jobs.
|
| 77 |
|
| 78 |
+
[Dataset and lab kit](https://huggingface.co/datasets/josefchen/flavourbench) ·
|
| 79 |
+
[Paper](https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf) ·
|
| 80 |
[Source](https://github.com/josefchen/flavourbench)
|
| 81 |
|
| 82 |
+
Josef Chen, Independent Researcher · Erim Hayretci, Imperial College London
|
| 83 |
|
| 84 |
```bibtex
|
| 85 |
@article{chen2026flavourbench,
|
app.py
CHANGED
|
@@ -4,6 +4,7 @@ import hashlib
|
|
| 4 |
import html
|
| 5 |
import json
|
| 6 |
import os
|
|
|
|
| 7 |
import tempfile
|
| 8 |
from datetime import UTC, datetime
|
| 9 |
from pathlib import Path
|
|
@@ -25,249 +26,427 @@ BUNDLE_PATH = Path(
|
|
| 25 |
)
|
| 26 |
)
|
| 27 |
|
| 28 |
-
|
| 29 |
-
|
| 30 |
|
| 31 |
CSS = """
|
| 32 |
:root {
|
| 33 |
-
--fb-accent: #
|
| 34 |
-
--fb-accent-soft: #
|
| 35 |
-
--fb-ink: #
|
| 36 |
-
--fb-muted: #
|
| 37 |
-
--fb-paper: #
|
| 38 |
-
--fb-
|
| 39 |
-
--fb-rule: #
|
| 40 |
-
--fb-code: #
|
| 41 |
}
|
| 42 |
.dark {
|
| 43 |
-
--fb-accent
|
| 44 |
-
--fb-
|
| 45 |
-
--fb-
|
| 46 |
-
--fb-
|
| 47 |
-
--fb-
|
| 48 |
-
--fb-
|
| 49 |
-
--fb-
|
|
|
|
| 50 |
}
|
|
|
|
| 51 |
body, .gradio-container {
|
| 52 |
background: var(--fb-paper) !important;
|
| 53 |
color: var(--fb-ink) !important;
|
| 54 |
font-family: "Geist", "Avenir Next", system-ui, sans-serif !important;
|
| 55 |
}
|
| 56 |
-
.gradio-container {
|
| 57 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
.fb-hero {
|
| 59 |
display: grid;
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
padding:
|
| 63 |
-
|
| 64 |
}
|
| 65 |
.fb-kicker {
|
| 66 |
color: var(--fb-accent);
|
| 67 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 68 |
-
font-size:
|
| 69 |
-
font-weight:
|
| 70 |
-
letter-spacing: .
|
| 71 |
text-transform: uppercase;
|
| 72 |
}
|
| 73 |
.fb-hero h1 {
|
| 74 |
color: var(--fb-ink);
|
| 75 |
-
font-size: clamp(
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
|
|
|
| 80 |
}
|
| 81 |
.fb-dek {
|
| 82 |
color: var(--fb-muted);
|
| 83 |
-
font-size:
|
| 84 |
-
line-height: 1.
|
| 85 |
margin: 0;
|
| 86 |
-
max-width:
|
| 87 |
-
}
|
| 88 |
-
.fb-byline {
|
| 89 |
-
display: flex;
|
| 90 |
-
flex-wrap: wrap;
|
| 91 |
-
gap: 10px 22px;
|
| 92 |
-
margin: 16px 0 0;
|
| 93 |
-
}
|
| 94 |
-
.fb-byline span {
|
| 95 |
-
color: var(--fb-ink);
|
| 96 |
-
display: flex;
|
| 97 |
-
flex-direction: column;
|
| 98 |
-
font-size: 14px;
|
| 99 |
-
font-weight: 650;
|
| 100 |
-
letter-spacing: .01em;
|
| 101 |
-
}
|
| 102 |
-
.fb-byline small {
|
| 103 |
-
color: var(--fb-muted);
|
| 104 |
-
font-size: 11px;
|
| 105 |
-
font-weight: 500;
|
| 106 |
-
letter-spacing: .04em;
|
| 107 |
-
margin-top: 2px;
|
| 108 |
}
|
| 109 |
.fb-stats {
|
|
|
|
| 110 |
display: grid;
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
margin-top:
|
|
|
|
| 114 |
}
|
| 115 |
-
.fb-stat { border-top: 2px solid var(--fb-rule); padding-top: 11px; }
|
| 116 |
.fb-stat strong {
|
| 117 |
color: var(--fb-ink);
|
| 118 |
display: block;
|
| 119 |
-
font-size:
|
|
|
|
| 120 |
letter-spacing: -.04em;
|
| 121 |
line-height: 1;
|
| 122 |
}
|
| 123 |
.fb-stat span {
|
| 124 |
color: var(--fb-muted);
|
| 125 |
display: block;
|
| 126 |
-
font-
|
|
|
|
|
|
|
| 127 |
margin-top: 7px;
|
| 128 |
text-transform: uppercase;
|
| 129 |
-
letter-spacing: .07em;
|
| 130 |
}
|
| 131 |
.fb-frontier {
|
| 132 |
align-self: end;
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
border-radius: 8px;
|
| 137 |
-
padding: 20px 22px 16px;
|
| 138 |
}
|
| 139 |
.fb-frontier-head {
|
| 140 |
align-items: baseline;
|
| 141 |
display: flex;
|
|
|
|
| 142 |
justify-content: space-between;
|
| 143 |
-
margin-bottom:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
}
|
| 145 |
-
.fb-frontier-head strong { font-size: 14px; }
|
| 146 |
-
.fb-frontier-head span { color: var(--fb-muted); font-size: 12px; }
|
| 147 |
.fb-forest-row {
|
| 148 |
align-items: center;
|
|
|
|
| 149 |
display: grid;
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
min-height:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
}
|
| 154 |
.fb-model {
|
| 155 |
color: var(--fb-ink);
|
| 156 |
font-size: 12px;
|
|
|
|
| 157 |
overflow: hidden;
|
| 158 |
text-overflow: ellipsis;
|
| 159 |
white-space: nowrap;
|
| 160 |
}
|
| 161 |
-
.fb-axis {
|
|
|
|
|
|
|
|
|
|
| 162 |
.fb-axis::before {
|
| 163 |
background: var(--fb-rule);
|
| 164 |
content: "";
|
| 165 |
-
height:
|
| 166 |
left: 0;
|
| 167 |
position: absolute;
|
| 168 |
right: 0;
|
| 169 |
-
top:
|
| 170 |
-
}
|
| 171 |
-
.fb-ci {
|
| 172 |
-
border-top: 2px solid var(--fb-muted);
|
| 173 |
-
height: 1px;
|
| 174 |
-
position: absolute;
|
| 175 |
-
top: 5px;
|
| 176 |
}
|
| 177 |
-
.fb-
|
| 178 |
-
background:
|
| 179 |
-
|
| 180 |
-
|
| 181 |
position: absolute;
|
| 182 |
-
top:
|
| 183 |
-
width: 1px;
|
| 184 |
}
|
| 185 |
-
.fb-ci::before { left: 0; }
|
| 186 |
-
.fb-ci::after { right: 0; }
|
| 187 |
.fb-point {
|
| 188 |
-
background: var(--fb-
|
| 189 |
-
|
|
|
|
| 190 |
position: absolute;
|
| 191 |
-
top:
|
| 192 |
-
|
|
|
|
| 193 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
| 194 |
.fb-number {
|
| 195 |
color: var(--fb-ink);
|
| 196 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 197 |
-
font-size:
|
|
|
|
| 198 |
text-align: right;
|
| 199 |
}
|
| 200 |
-
.fb-
|
| 201 |
color: var(--fb-muted);
|
|
|
|
| 202 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 203 |
-
font-size:
|
| 204 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 205 |
}
|
| 206 |
-
.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 207 |
.fb-section h2 {
|
| 208 |
color: var(--fb-ink);
|
| 209 |
-
font-size: 30px;
|
| 210 |
-
|
| 211 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 212 |
}
|
| 213 |
-
.fb-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 214 |
.fb-metric-grid {
|
|
|
|
|
|
|
| 215 |
display: grid;
|
| 216 |
grid-template-columns: repeat(3, 1fr);
|
| 217 |
-
|
| 218 |
-
margin: 8px 0 16px;
|
| 219 |
}
|
| 220 |
-
.fb-metric {
|
| 221 |
-
|
| 222 |
-
border: 1px solid var(--fb-rule);
|
| 223 |
-
|
| 224 |
-
padding: 16px;
|
| 225 |
}
|
|
|
|
| 226 |
.fb-metric small {
|
| 227 |
color: var(--fb-muted);
|
| 228 |
display: block;
|
| 229 |
-
font-
|
|
|
|
| 230 |
letter-spacing: .05em;
|
| 231 |
text-transform: uppercase;
|
| 232 |
}
|
| 233 |
-
.fb-metric strong {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 234 |
.fb-evidence {
|
| 235 |
-
|
| 236 |
-
border-
|
| 237 |
-
border-radius: 0 8px 8px 0;
|
| 238 |
color: var(--fb-ink);
|
| 239 |
-
line-height: 1.
|
| 240 |
-
padding:
|
| 241 |
}
|
|
|
|
| 242 |
.fb-evidence code, .fb-hash {
|
| 243 |
background: var(--fb-code);
|
| 244 |
color: var(--fb-muted);
|
| 245 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 246 |
font-size: 11px;
|
|
|
|
| 247 |
}
|
| 248 |
.fb-method {
|
| 249 |
display: grid;
|
| 250 |
-
|
| 251 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 252 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 253 |
.fb-footer {
|
| 254 |
border-top: 1px solid var(--fb-rule);
|
| 255 |
color: var(--fb-muted);
|
|
|
|
| 256 |
font-size: 12px;
|
| 257 |
-
|
| 258 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 259 |
}
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
.fb-hero { gap: 28px; padding-top: 34px; }
|
| 264 |
}
|
| 265 |
-
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 269 |
}
|
| 270 |
@media (prefers-reduced-motion: reduce) {
|
|
|
|
| 271 |
*, *::before, *::after { animation: none !important; transition: none !important; }
|
| 272 |
}
|
| 273 |
"""
|
|
@@ -341,6 +520,58 @@ TASK_LABEL_TO_ID = {
|
|
| 341 |
f"{row['task_id']} | {str(row['family']).replace('_', ' ')}": str(row["task_id"])
|
| 342 |
for row in TASKS
|
| 343 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 344 |
|
| 345 |
|
| 346 |
def _completion_diagnostic(model_id: str) -> dict[str, Any]:
|
|
@@ -378,61 +609,51 @@ def _completion_diagnostic(model_id: str) -> dict[str, Any]:
|
|
| 378 |
}
|
| 379 |
|
| 380 |
|
| 381 |
-
def _short(value: str) -> str:
|
| 382 |
-
return (
|
| 383 |
-
value.replace("GPT-5.6 ", "5.6 ")
|
| 384 |
-
.replace("Claude ", "")
|
| 385 |
-
.replace("DeepSeek ", "DS ")
|
| 386 |
-
.replace("Command ", "Cmd ")
|
| 387 |
-
)
|
| 388 |
-
|
| 389 |
-
|
| 390 |
def _frontier_html() -> str:
|
|
|
|
|
|
|
| 391 |
rows = []
|
| 392 |
-
for model in
|
| 393 |
score = float(model["flavourbench_score"])
|
| 394 |
-
|
| 395 |
-
|
| 396 |
-
|
| 397 |
-
|
| 398 |
rows.append(
|
| 399 |
"<div class='fb-forest-row'>"
|
| 400 |
-
f"<div class='fb-
|
| 401 |
-
f"{html.escape(
|
|
|
|
| 402 |
"<div class='fb-axis'>"
|
| 403 |
-
f"<span class='fb-
|
| 404 |
-
f"<span class='fb-point' style='left:{
|
| 405 |
"</div>"
|
| 406 |
f"<div class='fb-number'>{score:.1f}</div>"
|
| 407 |
-
f"<div class='fb-group'>G{group if group is not None else '-'}</div>"
|
| 408 |
"</div>"
|
| 409 |
)
|
| 410 |
-
return "".join(rows)
|
|
|
|
|
|
|
|
|
|
| 411 |
|
| 412 |
|
| 413 |
def _hero_html() -> str:
|
| 414 |
inference = BUNDLE["analysis"]["inference"]
|
| 415 |
-
panel_phrase = f" across {PANEL_COUNT} collection panels" if PANEL_COUNT > 1 else ""
|
| 416 |
return f"""
|
| 417 |
<div class="fb-shell fb-hero">
|
| 418 |
<section>
|
| 419 |
-
<div class="fb-kicker">
|
| 420 |
-
<h1>
|
| 421 |
-
<p class="fb-dek">
|
| 422 |
-
with anchor-clustered uncertainty and inspectable responses.</p>
|
| 423 |
-
<div class="fb-byline">
|
| 424 |
-
<span>Josef Chen<small>Independent Researcher</small></span>
|
| 425 |
-
<span>Erim Hayretci<small>Imperial College London</small></span>
|
| 426 |
-
</div>
|
| 427 |
<div class="fb-stats">
|
| 428 |
-
<div class="fb-stat"><strong>{MODEL_COUNT}</strong><span>
|
| 429 |
-
<div class="fb-stat"><strong>{TASK_COUNT}</strong><span>tasks</span></div>
|
| 430 |
-
<div class="fb-stat"><strong>{PRIMARY_COUNT:,}</strong><span>
|
| 431 |
-
<div class="fb-stat"><strong>{inference["pairwise_hypotheses"]}</strong><span>
|
| 432 |
</div>
|
| 433 |
</section>
|
| 434 |
-
<section class="fb-frontier" aria-label="
|
| 435 |
-
<div class="fb-frontier-head"><strong>
|
| 436 |
{_frontier_html()}
|
| 437 |
</section>
|
| 438 |
</div>
|
|
@@ -447,18 +668,48 @@ def _leaderboard_frame() -> pd.DataFrame:
|
|
| 447 |
rows.append(
|
| 448 |
{
|
| 449 |
"Rank": model["point_estimate_rank"],
|
| 450 |
-
"Group": model["statistical_rank_group"],
|
| 451 |
"Model": model["model_name"],
|
| 452 |
-
"Score": round(float(model["flavourbench_score"]), 2),
|
| 453 |
"Simultaneous 95%": f"{ci[0]:.2f} to {ci[1]:.2f}",
|
|
|
|
| 454 |
"Rank 95%": f"{rank_ci[0]} to {rank_ci[1]}",
|
| 455 |
"Cells": f"{model['coverage']['valid_scored']}/{TASK_COUNT}",
|
| 456 |
-
"Backend": model["execution_backend"],
|
| 457 |
}
|
| 458 |
)
|
| 459 |
return pd.DataFrame(rows)
|
| 460 |
|
| 461 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 462 |
def _model_detail(model_name: str) -> tuple[str, pd.DataFrame]:
|
| 463 |
model = MODEL_BY_NAME[model_name]
|
| 464 |
rank_interval = model["bootstrap_rank_95_interval"]
|
|
@@ -614,6 +865,55 @@ def _training_reward_api(task_id: str, completion: str) -> dict[str, Any]:
|
|
| 614 |
}
|
| 615 |
|
| 616 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 617 |
def _score_upload(
|
| 618 |
artifact_path: str | None,
|
| 619 |
model_name: str,
|
|
@@ -689,62 +989,56 @@ def _score_upload(
|
|
| 689 |
|
| 690 |
theme = gr.themes.Base(
|
| 691 |
primary_hue=gr.themes.Color(
|
| 692 |
-
c50="#
|
| 693 |
-
c100="#
|
| 694 |
-
c200="#
|
| 695 |
-
c300="#
|
| 696 |
-
c400="#
|
| 697 |
-
c500=
|
| 698 |
-
c600="#
|
| 699 |
-
c700="#
|
| 700 |
-
c800="#
|
| 701 |
-
c900="#
|
| 702 |
-
c950="#
|
| 703 |
),
|
| 704 |
-
neutral_hue="
|
| 705 |
font=gr.themes.GoogleFont("Geist", weights=(400, 500, 600, 700)),
|
| 706 |
font_mono=gr.themes.GoogleFont("IBM Plex Mono", weights=(400, 600)),
|
| 707 |
).set(
|
| 708 |
-
body_background_fill="#
|
| 709 |
-
block_background_fill="#
|
| 710 |
-
block_border_width="
|
| 711 |
block_label_text_weight="600",
|
| 712 |
-
button_primary_background_fill=
|
| 713 |
-
button_primary_background_fill_hover=
|
| 714 |
)
|
| 715 |
|
| 716 |
|
| 717 |
-
with gr.Blocks(title="FlavourBench |
|
| 718 |
gr.HTML(_hero_html())
|
| 719 |
with gr.Tabs():
|
| 720 |
-
with gr.Tab("
|
| 721 |
gr.HTML(
|
| 722 |
-
"""
|
| 723 |
<div class="fb-section">
|
| 724 |
-
<h2>The
|
| 725 |
-
<p>
|
| 726 |
</div>
|
| 727 |
"""
|
| 728 |
)
|
| 729 |
-
gr.
|
| 730 |
-
value=_leaderboard_frame(),
|
| 731 |
-
interactive=False,
|
| 732 |
-
wrap=True,
|
| 733 |
-
show_search="filter",
|
| 734 |
-
show_row_numbers=False,
|
| 735 |
-
column_widths=[55, 55, 245, 80, 180, 110, 125, 120],
|
| 736 |
-
)
|
| 737 |
gr.Markdown(
|
| 738 |
-
"
|
| 739 |
-
"
|
|
|
|
| 740 |
)
|
| 741 |
|
| 742 |
-
with gr.Tab("
|
| 743 |
gr.HTML(
|
| 744 |
"""
|
| 745 |
<div class="fb-section">
|
| 746 |
-
<h2>
|
| 747 |
-
<p>
|
| 748 |
</div>
|
| 749 |
"""
|
| 750 |
)
|
|
@@ -765,14 +1059,15 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
|
|
| 765 |
_model_detail,
|
| 766 |
inputs=model_selector,
|
| 767 |
outputs=[model_summary, family_table],
|
|
|
|
| 768 |
)
|
| 769 |
|
| 770 |
-
with gr.Tab("
|
| 771 |
gr.HTML(
|
| 772 |
"""
|
| 773 |
<div class="fb-section">
|
| 774 |
<h2>Open one scored decision</h2>
|
| 775 |
-
<p>
|
| 776 |
</div>
|
| 777 |
"""
|
| 778 |
)
|
|
@@ -791,36 +1086,120 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
|
|
| 791 |
filterable=True,
|
| 792 |
scale=2,
|
| 793 |
)
|
| 794 |
-
inspect_task = gr.Button("Inspect", variant="primary", scale=0)
|
| 795 |
initial = _task_detail(MODEL_NAMES[0], next(iter(TASK_LABEL_TO_ID)))
|
| 796 |
task_status = gr.HTML(initial[0])
|
| 797 |
-
prompt = gr.Textbox(value=initial[1], label="Exact prompt", lines=
|
| 798 |
-
choices = gr.JSON(value=initial[2], label="
|
| 799 |
score_map = gr.Dataframe(
|
| 800 |
value=initial[3],
|
| 801 |
-
label="Top 12 of 56
|
| 802 |
interactive=False,
|
| 803 |
wrap=True,
|
| 804 |
show_row_numbers=False,
|
| 805 |
)
|
| 806 |
answer = gr.Markdown(value=initial[4], label="Model response")
|
| 807 |
-
provenance = gr.Markdown(value=initial[5], label="
|
| 808 |
inspect_task.click(
|
| 809 |
_task_detail,
|
| 810 |
inputs=[task_model, task_selector],
|
| 811 |
outputs=[task_status, prompt, choices, score_map, answer, provenance],
|
|
|
|
| 812 |
)
|
| 813 |
|
| 814 |
-
with gr.Tab("
|
| 815 |
gr.HTML(
|
| 816 |
f"""
|
| 817 |
<div class="fb-section">
|
| 818 |
-
<h2>
|
| 819 |
-
<p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 820 |
</div>
|
| 821 |
-
|
| 822 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 823 |
</div>
|
|
|
|
| 824 |
"""
|
| 825 |
)
|
| 826 |
with gr.Row():
|
|
@@ -845,7 +1224,7 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
|
|
| 845 |
file_types=[".jsonl", ".json"],
|
| 846 |
type="filepath",
|
| 847 |
)
|
| 848 |
-
score_upload = gr.Button("Score artifact", variant="primary")
|
| 849 |
lab_summary = gr.Markdown()
|
| 850 |
lab_rows = gr.Dataframe(
|
| 851 |
interactive=False,
|
|
@@ -863,26 +1242,22 @@ with gr.Blocks(title="FlavourBench | Executable culinary evaluation") as demo:
|
|
| 863 |
)
|
| 864 |
gr.Markdown(
|
| 865 |
"""
|
| 866 |
-
The accepted JSONL contract is one object per task:
|
| 867 |
-
|
| 868 |
```json
|
| 869 |
{"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
|
| 870 |
```
|
| 871 |
|
| 872 |
-
|
| 873 |
-
|
| 874 |
-
|
| 875 |
-
also provides a local runner, scorer, schemas, and TRL recipes. Local reward lookup remains the
|
| 876 |
-
recommended path for high-throughput training.
|
| 877 |
"""
|
| 878 |
)
|
| 879 |
|
| 880 |
-
with gr.Tab("
|
| 881 |
gr.HTML(
|
| 882 |
f"""
|
| 883 |
<div class="fb-section">
|
| 884 |
-
<h2>
|
| 885 |
-
<p>Query any
|
| 886 |
</div>
|
| 887 |
"""
|
| 888 |
)
|
|
@@ -899,34 +1274,45 @@ recommended path for high-throughput training.
|
|
| 899 |
label="Second model",
|
| 900 |
filterable=True,
|
| 901 |
)
|
| 902 |
-
compare = gr.Button("Compare", variant="primary", scale=0)
|
| 903 |
pair_result = gr.HTML(_pair_detail(MODEL_NAMES[0], MODEL_NAMES[1]))
|
| 904 |
-
compare.click(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 905 |
|
| 906 |
-
with gr.Tab("Method
|
| 907 |
gr.HTML(
|
| 908 |
f"""
|
| 909 |
<div class="fb-section">
|
| 910 |
-
<h2>One
|
| 911 |
-
<p>The Space makes no
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 912 |
</div>
|
| 913 |
<div class="fb-method">
|
| 914 |
<div>
|
| 915 |
<h3>Scoring contract</h3>
|
| 916 |
-
<p>
|
| 917 |
-
|
| 918 |
-
constraint tasks. The release uses a complete 27-by-534 matrix: every ranked
|
| 919 |
-
model contributes one valid response to every scored task.</p>
|
| 920 |
<h3>Inference</h3>
|
| 921 |
<p>Results use {INDEPENDENT_CLUSTER_COUNT:,} ingredient-anchor clusters,
|
| 922 |
-
50,000
|
| 923 |
-
|
| 924 |
-
|
|
|
|
|
|
|
|
|
|
| 925 |
</div>
|
| 926 |
<aside class="fb-evidence">
|
| 927 |
-
<strong>
|
| 928 |
<span class="fb-hash">{BUNDLE["release_artifact_sha256"]}</span><br><br>
|
| 929 |
-
{MODEL_COUNT}
|
| 930 |
</aside>
|
| 931 |
</div>
|
| 932 |
"""
|
|
@@ -936,12 +1322,9 @@ recommended path for high-throughput training.
|
|
| 936 |
```bash
|
| 937 |
git clone https://github.com/josefchen/flavourbench.git
|
| 938 |
cd flavourbench
|
| 939 |
-
pip install -e '.[dev]'
|
| 940 |
-
|
| 941 |
-
make -C paper -f Makefile.powered arxiv
|
| 942 |
```
|
| 943 |
-
|
| 944 |
-
[Paper](https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf) | [Dataset](https://huggingface.co/datasets/josefchen/flavourbench) | [Source](https://github.com/josefchen/flavourbench)
|
| 945 |
"""
|
| 946 |
)
|
| 947 |
|
|
@@ -967,7 +1350,8 @@ make -C paper -f Makefile.powered arxiv
|
|
| 967 |
gr.HTML(
|
| 968 |
"""
|
| 969 |
<div class="fb-shell fb-footer">
|
| 970 |
-
FlavourBench
|
|
|
|
| 971 |
</div>
|
| 972 |
"""
|
| 973 |
)
|
|
|
|
| 4 |
import html
|
| 5 |
import json
|
| 6 |
import os
|
| 7 |
+
import shlex
|
| 8 |
import tempfile
|
| 9 |
from datetime import UTC, datetime
|
| 10 |
from pathlib import Path
|
|
|
|
| 26 |
)
|
| 27 |
)
|
| 28 |
|
| 29 |
+
RUST = "#B54135"
|
| 30 |
+
INK = "#171815"
|
| 31 |
|
| 32 |
CSS = """
|
| 33 |
:root {
|
| 34 |
+
--fb-accent: #B54135;
|
| 35 |
+
--fb-accent-soft: #F1DDD8;
|
| 36 |
+
--fb-ink: #171815;
|
| 37 |
+
--fb-muted: #666960;
|
| 38 |
+
--fb-paper: #F4F3EE;
|
| 39 |
+
--fb-paper-raised: #FAF9F5;
|
| 40 |
+
--fb-rule: #C9CBC4;
|
| 41 |
+
--fb-code: #E8E7E1;
|
| 42 |
}
|
| 43 |
.dark {
|
| 44 |
+
--fb-accent: #EF796D;
|
| 45 |
+
--fb-accent-soft: #422723;
|
| 46 |
+
--fb-ink: #F0EFE9;
|
| 47 |
+
--fb-muted: #A9ACA3;
|
| 48 |
+
--fb-paper: #171815;
|
| 49 |
+
--fb-paper-raised: #20211E;
|
| 50 |
+
--fb-rule: #41433D;
|
| 51 |
+
--fb-code: #2A2B27;
|
| 52 |
}
|
| 53 |
+
html { scroll-behavior: smooth; }
|
| 54 |
body, .gradio-container {
|
| 55 |
background: var(--fb-paper) !important;
|
| 56 |
color: var(--fb-ink) !important;
|
| 57 |
font-family: "Geist", "Avenir Next", system-ui, sans-serif !important;
|
| 58 |
}
|
| 59 |
+
.gradio-container {
|
| 60 |
+
max-width: none !important;
|
| 61 |
+
padding: 0 !important;
|
| 62 |
+
}
|
| 63 |
+
.gradio-container > main { padding: 0 !important; }
|
| 64 |
+
.fb-shell {
|
| 65 |
+
box-sizing: border-box;
|
| 66 |
+
margin: 0 auto;
|
| 67 |
+
max-width: 1440px;
|
| 68 |
+
padding-left: clamp(22px, 4vw, 62px);
|
| 69 |
+
padding-right: clamp(22px, 4vw, 62px);
|
| 70 |
+
}
|
| 71 |
.fb-hero {
|
| 72 |
display: grid;
|
| 73 |
+
gap: clamp(40px, 6vw, 96px);
|
| 74 |
+
grid-template-columns: minmax(360px, .82fr) minmax(540px, 1.18fr);
|
| 75 |
+
padding-bottom: 42px;
|
| 76 |
+
padding-top: clamp(46px, 6vw, 86px);
|
| 77 |
}
|
| 78 |
.fb-kicker {
|
| 79 |
color: var(--fb-accent);
|
| 80 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 81 |
+
font-size: 11px;
|
| 82 |
+
font-weight: 600;
|
| 83 |
+
letter-spacing: .13em;
|
| 84 |
text-transform: uppercase;
|
| 85 |
}
|
| 86 |
.fb-hero h1 {
|
| 87 |
color: var(--fb-ink);
|
| 88 |
+
font-size: clamp(56px, 6vw, 92px);
|
| 89 |
+
font-weight: 650;
|
| 90 |
+
letter-spacing: -.065em;
|
| 91 |
+
line-height: .91;
|
| 92 |
+
margin: 18px 0 23px;
|
| 93 |
+
max-width: 740px;
|
| 94 |
}
|
| 95 |
.fb-dek {
|
| 96 |
color: var(--fb-muted);
|
| 97 |
+
font-size: clamp(17px, 1.6vw, 21px);
|
| 98 |
+
line-height: 1.46;
|
| 99 |
margin: 0;
|
| 100 |
+
max-width: 570px;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
}
|
| 102 |
.fb-stats {
|
| 103 |
+
border-top: 1px solid var(--fb-ink);
|
| 104 |
display: grid;
|
| 105 |
+
gap: 20px;
|
| 106 |
+
grid-template-columns: repeat(4, minmax(88px, 1fr));
|
| 107 |
+
margin-top: 36px;
|
| 108 |
+
padding-top: 13px;
|
| 109 |
}
|
|
|
|
| 110 |
.fb-stat strong {
|
| 111 |
color: var(--fb-ink);
|
| 112 |
display: block;
|
| 113 |
+
font-size: 25px;
|
| 114 |
+
font-weight: 620;
|
| 115 |
letter-spacing: -.04em;
|
| 116 |
line-height: 1;
|
| 117 |
}
|
| 118 |
.fb-stat span {
|
| 119 |
color: var(--fb-muted);
|
| 120 |
display: block;
|
| 121 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 122 |
+
font-size: 10px;
|
| 123 |
+
letter-spacing: .06em;
|
| 124 |
margin-top: 7px;
|
| 125 |
text-transform: uppercase;
|
|
|
|
| 126 |
}
|
| 127 |
.fb-frontier {
|
| 128 |
align-self: end;
|
| 129 |
+
border-top: 1px solid var(--fb-ink);
|
| 130 |
+
min-width: 0;
|
| 131 |
+
padding-top: 12px;
|
|
|
|
|
|
|
| 132 |
}
|
| 133 |
.fb-frontier-head {
|
| 134 |
align-items: baseline;
|
| 135 |
display: flex;
|
| 136 |
+
gap: 16px;
|
| 137 |
justify-content: space-between;
|
| 138 |
+
margin-bottom: 15px;
|
| 139 |
+
}
|
| 140 |
+
.fb-frontier-head strong { font-size: 15px; font-weight: 620; }
|
| 141 |
+
.fb-frontier-head span {
|
| 142 |
+
color: var(--fb-muted);
|
| 143 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 144 |
+
font-size: 10px;
|
| 145 |
+
text-align: right;
|
| 146 |
}
|
|
|
|
|
|
|
| 147 |
.fb-forest-row {
|
| 148 |
align-items: center;
|
| 149 |
+
border-bottom: 1px solid var(--fb-rule);
|
| 150 |
display: grid;
|
| 151 |
+
gap: 12px;
|
| 152 |
+
grid-template-columns: 22px minmax(130px, 190px) 1fr 47px;
|
| 153 |
+
min-height: 36px;
|
| 154 |
+
}
|
| 155 |
+
.fb-place {
|
| 156 |
+
color: var(--fb-muted);
|
| 157 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 158 |
+
font-size: 10px;
|
| 159 |
}
|
| 160 |
.fb-model {
|
| 161 |
color: var(--fb-ink);
|
| 162 |
font-size: 12px;
|
| 163 |
+
font-weight: 560;
|
| 164 |
overflow: hidden;
|
| 165 |
text-overflow: ellipsis;
|
| 166 |
white-space: nowrap;
|
| 167 |
}
|
| 168 |
+
.fb-axis {
|
| 169 |
+
height: 10px;
|
| 170 |
+
position: relative;
|
| 171 |
+
}
|
| 172 |
.fb-axis::before {
|
| 173 |
background: var(--fb-rule);
|
| 174 |
content: "";
|
| 175 |
+
height: 2px;
|
| 176 |
left: 0;
|
| 177 |
position: absolute;
|
| 178 |
right: 0;
|
| 179 |
+
top: 4px;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 180 |
}
|
| 181 |
+
.fb-bar {
|
| 182 |
+
background: #A7ABA4;
|
| 183 |
+
height: 4px;
|
| 184 |
+
left: 0;
|
| 185 |
position: absolute;
|
| 186 |
+
top: 3px;
|
|
|
|
| 187 |
}
|
|
|
|
|
|
|
| 188 |
.fb-point {
|
| 189 |
+
background: var(--fb-ink);
|
| 190 |
+
border-radius: 50%;
|
| 191 |
+
height: 10px;
|
| 192 |
position: absolute;
|
| 193 |
+
top: 0;
|
| 194 |
+
transform: translateX(-50%);
|
| 195 |
+
width: 10px;
|
| 196 |
}
|
| 197 |
+
.fb-forest-row:first-of-type .fb-bar,
|
| 198 |
+
.fb-forest-row:first-of-type .fb-point { background: var(--fb-accent); }
|
| 199 |
+
.fb-forest-row:first-of-type .fb-model,
|
| 200 |
+
.fb-forest-row:first-of-type .fb-number { color: var(--fb-accent); }
|
| 201 |
.fb-number {
|
| 202 |
color: var(--fb-ink);
|
| 203 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 204 |
+
font-size: 12px;
|
| 205 |
+
font-weight: 600;
|
| 206 |
text-align: right;
|
| 207 |
}
|
| 208 |
+
.fb-chart-foot {
|
| 209 |
color: var(--fb-muted);
|
| 210 |
+
display: flex;
|
| 211 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 212 |
+
font-size: 9px;
|
| 213 |
+
justify-content: space-between;
|
| 214 |
+
margin: 9px 59px 0 224px;
|
| 215 |
+
}
|
| 216 |
+
.tab-wrapper {
|
| 217 |
+
border-bottom: 1px solid var(--fb-rule);
|
| 218 |
+
border-top: 1px solid var(--fb-rule);
|
| 219 |
}
|
| 220 |
+
.tab-nav {
|
| 221 |
+
margin: 0 auto !important;
|
| 222 |
+
max-width: 1316px !important;
|
| 223 |
+
padding-left: clamp(22px, 4vw, 62px) !important;
|
| 224 |
+
padding-right: clamp(22px, 4vw, 62px) !important;
|
| 225 |
+
}
|
| 226 |
+
.tab-nav button {
|
| 227 |
+
border: 0 !important;
|
| 228 |
+
color: var(--fb-muted) !important;
|
| 229 |
+
font-size: 13px !important;
|
| 230 |
+
padding: 15px 0 13px !important;
|
| 231 |
+
margin-right: 30px !important;
|
| 232 |
+
}
|
| 233 |
+
.tab-nav button.selected {
|
| 234 |
+
border-bottom: 2px solid var(--fb-accent) !important;
|
| 235 |
+
color: var(--fb-ink) !important;
|
| 236 |
+
}
|
| 237 |
+
.overflow-menu { display: none !important; }
|
| 238 |
+
.tab-container.visually-hidden { display: none !important; }
|
| 239 |
+
.tabitem {
|
| 240 |
+
box-sizing: border-box;
|
| 241 |
+
margin: 0 auto !important;
|
| 242 |
+
max-width: 1440px !important;
|
| 243 |
+
padding: 0 clamp(22px, 4vw, 62px) !important;
|
| 244 |
+
}
|
| 245 |
+
.fb-section { margin: 36px 0 18px; }
|
| 246 |
.fb-section h2 {
|
| 247 |
color: var(--fb-ink);
|
| 248 |
+
font-size: clamp(30px, 3vw, 45px);
|
| 249 |
+
font-weight: 620;
|
| 250 |
+
letter-spacing: -.045em;
|
| 251 |
+
line-height: 1.02;
|
| 252 |
+
margin: 0 0 9px;
|
| 253 |
+
}
|
| 254 |
+
.fb-section p {
|
| 255 |
+
color: var(--fb-muted);
|
| 256 |
+
font-size: 15px;
|
| 257 |
+
line-height: 1.5;
|
| 258 |
+
margin: 0;
|
| 259 |
+
max-width: 72ch;
|
| 260 |
}
|
| 261 |
+
.fb-step-line {
|
| 262 |
+
border-bottom: 1px solid var(--fb-rule);
|
| 263 |
+
border-top: 1px solid var(--fb-rule);
|
| 264 |
+
display: grid;
|
| 265 |
+
grid-template-columns: repeat(3, 1fr);
|
| 266 |
+
margin: 8px 0 26px;
|
| 267 |
+
}
|
| 268 |
+
.fb-step { padding: 18px 22px 20px 0; }
|
| 269 |
+
.fb-step + .fb-step { border-left: 1px solid var(--fb-rule); padding-left: 22px; }
|
| 270 |
+
.fb-step b {
|
| 271 |
+
color: var(--fb-accent);
|
| 272 |
+
display: block;
|
| 273 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 274 |
+
font-size: 10px;
|
| 275 |
+
letter-spacing: .08em;
|
| 276 |
+
margin-bottom: 7px;
|
| 277 |
+
text-transform: uppercase;
|
| 278 |
+
}
|
| 279 |
+
.fb-step strong { display: block; font-size: 17px; margin-bottom: 4px; }
|
| 280 |
+
.fb-step span { color: var(--fb-muted); font-size: 13px; line-height: 1.45; }
|
| 281 |
.fb-metric-grid {
|
| 282 |
+
border-bottom: 1px solid var(--fb-rule);
|
| 283 |
+
border-top: 1px solid var(--fb-rule);
|
| 284 |
display: grid;
|
| 285 |
grid-template-columns: repeat(3, 1fr);
|
| 286 |
+
margin: 10px 0 22px;
|
|
|
|
| 287 |
}
|
| 288 |
+
.fb-metric { min-width: 0; padding: 19px 22px 21px 0; }
|
| 289 |
+
.fb-metric:nth-child(3n+2), .fb-metric:nth-child(3n+3) {
|
| 290 |
+
border-left: 1px solid var(--fb-rule);
|
| 291 |
+
padding-left: 22px;
|
|
|
|
| 292 |
}
|
| 293 |
+
.fb-metric:nth-child(n+4) { border-top: 1px solid var(--fb-rule); }
|
| 294 |
.fb-metric small {
|
| 295 |
color: var(--fb-muted);
|
| 296 |
display: block;
|
| 297 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 298 |
+
font-size: 10px;
|
| 299 |
letter-spacing: .05em;
|
| 300 |
text-transform: uppercase;
|
| 301 |
}
|
| 302 |
+
.fb-metric strong {
|
| 303 |
+
color: var(--fb-ink);
|
| 304 |
+
display: block;
|
| 305 |
+
font-size: 24px;
|
| 306 |
+
font-weight: 620;
|
| 307 |
+
margin-top: 5px;
|
| 308 |
+
overflow-wrap: anywhere;
|
| 309 |
+
}
|
| 310 |
.fb-evidence {
|
| 311 |
+
border-bottom: 1px solid var(--fb-rule);
|
| 312 |
+
border-top: 1px solid var(--fb-rule);
|
|
|
|
| 313 |
color: var(--fb-ink);
|
| 314 |
+
line-height: 1.52;
|
| 315 |
+
padding: 15px 0;
|
| 316 |
}
|
| 317 |
+
.fb-evidence strong:first-child { color: var(--fb-accent); }
|
| 318 |
.fb-evidence code, .fb-hash {
|
| 319 |
background: var(--fb-code);
|
| 320 |
color: var(--fb-muted);
|
| 321 |
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 322 |
font-size: 11px;
|
| 323 |
+
overflow-wrap: anywhere;
|
| 324 |
}
|
| 325 |
.fb-method {
|
| 326 |
display: grid;
|
| 327 |
+
gap: clamp(30px, 6vw, 86px);
|
| 328 |
+
grid-template-columns: 1.25fr .75fr;
|
| 329 |
+
}
|
| 330 |
+
.fb-method h3 { font-size: 19px; margin: 23px 0 6px; }
|
| 331 |
+
.fb-method p { color: var(--fb-muted); line-height: 1.55; }
|
| 332 |
+
.fb-command-note {
|
| 333 |
+
color: var(--fb-muted);
|
| 334 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 335 |
+
font-size: 10px;
|
| 336 |
+
line-height: 1.5;
|
| 337 |
+
}
|
| 338 |
+
.fb-table-wrap {
|
| 339 |
+
overflow-x: auto;
|
| 340 |
+
width: 100%;
|
| 341 |
}
|
| 342 |
+
.fb-table {
|
| 343 |
+
border: 0 !important;
|
| 344 |
+
color: var(--fb-ink);
|
| 345 |
+
font-family: "IBM Plex Mono", ui-monospace, monospace;
|
| 346 |
+
font-size: 12px;
|
| 347 |
+
min-width: 860px;
|
| 348 |
+
width: 100%;
|
| 349 |
+
}
|
| 350 |
+
.fb-table caption {
|
| 351 |
+
height: 1px;
|
| 352 |
+
overflow: hidden;
|
| 353 |
+
position: absolute;
|
| 354 |
+
width: 1px;
|
| 355 |
+
}
|
| 356 |
+
.fb-table th {
|
| 357 |
+
border-left: 0 !important;
|
| 358 |
+
border-right: 0 !important;
|
| 359 |
+
border-top: 0 !important;
|
| 360 |
+
border-bottom: 1px solid var(--fb-ink);
|
| 361 |
+
color: var(--fb-muted);
|
| 362 |
+
font-size: 10px;
|
| 363 |
+
font-weight: 600;
|
| 364 |
+
letter-spacing: .05em;
|
| 365 |
+
padding: 10px 12px 11px 0;
|
| 366 |
+
text-align: left;
|
| 367 |
+
text-transform: uppercase;
|
| 368 |
+
}
|
| 369 |
+
.fb-table td {
|
| 370 |
+
border-left: 0 !important;
|
| 371 |
+
border-right: 0 !important;
|
| 372 |
+
border-top: 0 !important;
|
| 373 |
+
border-bottom: 1px solid var(--fb-rule);
|
| 374 |
+
padding: 11px 12px 11px 0;
|
| 375 |
+
vertical-align: top;
|
| 376 |
+
}
|
| 377 |
+
.fb-table .fb-score-cell { font-weight: 600; }
|
| 378 |
+
.fb-table tbody tr:first-child .fb-rank-cell,
|
| 379 |
+
.fb-table tbody tr:first-child .fb-score-cell { color: var(--fb-accent); }
|
| 380 |
+
.fb-table tbody tr:hover { background: color-mix(in srgb, var(--fb-ink) 3%, transparent); }
|
| 381 |
.fb-footer {
|
| 382 |
border-top: 1px solid var(--fb-rule);
|
| 383 |
color: var(--fb-muted);
|
| 384 |
+
display: flex;
|
| 385 |
font-size: 12px;
|
| 386 |
+
justify-content: space-between;
|
| 387 |
+
margin-top: 54px;
|
| 388 |
+
padding-bottom: 30px;
|
| 389 |
+
padding-top: 18px;
|
| 390 |
+
}
|
| 391 |
+
.fb-footer a { color: var(--fb-ink); text-decoration: none; }
|
| 392 |
+
.fb-footer a:hover { color: var(--fb-accent); text-decoration: underline; }
|
| 393 |
+
.gradio-container footer a, .gradio-container footer button { color: var(--fb-muted) !important; }
|
| 394 |
+
.block, .form, .gradio-dataframe, .gradio-json, .gradio-textbox {
|
| 395 |
+
border-radius: 0 !important;
|
| 396 |
+
box-shadow: none !important;
|
| 397 |
+
}
|
| 398 |
+
.block:not(.gradio-dataframe):not(.gradio-json):not(.gradio-textbox) {
|
| 399 |
+
border-color: transparent !important;
|
| 400 |
+
}
|
| 401 |
+
button.primary {
|
| 402 |
+
background: var(--fb-ink) !important;
|
| 403 |
+
border: 1px solid var(--fb-ink) !important;
|
| 404 |
+
border-radius: 0 !important;
|
| 405 |
+
color: var(--fb-paper) !important;
|
| 406 |
+
min-height: 44px !important;
|
| 407 |
+
}
|
| 408 |
+
button.primary:hover { background: var(--fb-accent) !important; border-color: var(--fb-accent) !important; }
|
| 409 |
+
input, textarea, select {
|
| 410 |
+
border-radius: 0 !important;
|
| 411 |
+
box-shadow: none !important;
|
| 412 |
}
|
| 413 |
+
input:focus, textarea:focus, select:focus, button:focus-visible, a:focus-visible {
|
| 414 |
+
outline: 2px solid var(--fb-accent) !important;
|
| 415 |
+
outline-offset: 2px !important;
|
|
|
|
| 416 |
}
|
| 417 |
+
table { border-collapse: collapse !important; }
|
| 418 |
+
th { background: var(--fb-paper) !important; }
|
| 419 |
+
pre, code { border-radius: 0 !important; }
|
| 420 |
+
@media (max-width: 1040px) {
|
| 421 |
+
.fb-hero { grid-template-columns: 1fr; }
|
| 422 |
+
.fb-frontier { max-width: 820px; }
|
| 423 |
+
.fb-method { grid-template-columns: 1fr; }
|
| 424 |
+
}
|
| 425 |
+
@media (max-width: 720px) {
|
| 426 |
+
.fb-hero { gap: 38px; padding-bottom: 32px; }
|
| 427 |
+
.fb-hero h1 { font-size: clamp(52px, 15vw, 72px); }
|
| 428 |
+
.fb-stats { gap: 16px 10px; grid-template-columns: repeat(2, 1fr); }
|
| 429 |
+
.fb-forest-row { gap: 8px; grid-template-columns: 19px minmax(96px, 142px) 1fr 42px; }
|
| 430 |
+
.fb-model { font-size: 11px; }
|
| 431 |
+
.fb-chart-foot { margin-left: 133px; margin-right: 50px; }
|
| 432 |
+
.fb-chart-foot span:nth-child(2) { display: none; }
|
| 433 |
+
.fb-step-line { grid-template-columns: 1fr; }
|
| 434 |
+
.fb-step + .fb-step { border-left: 0; border-top: 1px solid var(--fb-rule); padding-left: 0; }
|
| 435 |
+
.fb-metric-grid { grid-template-columns: repeat(2, 1fr); }
|
| 436 |
+
.fb-metric:nth-child(3n+2), .fb-metric:nth-child(3n+3) { border-left: 0; padding-left: 0; }
|
| 437 |
+
.fb-metric:nth-child(even) { border-left: 1px solid var(--fb-rule); padding-left: 18px; }
|
| 438 |
+
.fb-metric:nth-child(n+3) { border-top: 1px solid var(--fb-rule); }
|
| 439 |
+
.tab-nav { overflow-x: auto !important; }
|
| 440 |
+
.tab-nav button { margin-right: 22px !important; white-space: nowrap !important; }
|
| 441 |
+
.fb-footer { display: block; }
|
| 442 |
+
.fb-footer span { display: block; margin-top: 7px; }
|
| 443 |
+
}
|
| 444 |
+
@media (max-width: 430px) {
|
| 445 |
+
.fb-forest-row { grid-template-columns: 18px minmax(82px, 114px) 1fr 39px; }
|
| 446 |
+
.fb-chart-foot { margin-left: 109px; margin-right: 47px; }
|
| 447 |
}
|
| 448 |
@media (prefers-reduced-motion: reduce) {
|
| 449 |
+
html { scroll-behavior: auto; }
|
| 450 |
*, *::before, *::after { animation: none !important; transition: none !important; }
|
| 451 |
}
|
| 452 |
"""
|
|
|
|
| 520 |
f"{row['task_id']} | {str(row['family']).replace('_', ' ')}": str(row["task_id"])
|
| 521 |
for row in TASKS
|
| 522 |
}
|
| 523 |
+
LAB_TASK_LABEL_TO_ID = {
|
| 524 |
+
f"{row['task_id']} | {str(row['family']).replace('_', ' ')} | {row['lab_split']}": str(
|
| 525 |
+
row["task_id"]
|
| 526 |
+
)
|
| 527 |
+
for row in LAB_TASKS
|
| 528 |
+
}
|
| 529 |
+
|
| 530 |
+
|
| 531 |
+
def _lab_name(row: dict[str, Any]) -> str:
|
| 532 |
+
model_name = str(row["model_name"])
|
| 533 |
+
prefixes = (
|
| 534 |
+
("SpaceXAI:", "xAI"),
|
| 535 |
+
("Anthropic:", "Anthropic"),
|
| 536 |
+
("Claude ", "Anthropic"),
|
| 537 |
+
("MoonshotAI:", "Kimi"),
|
| 538 |
+
("ByteDance Seed:", "ByteDance"),
|
| 539 |
+
("Thinking Machines:", "Thinking Machines"),
|
| 540 |
+
("MiniMax:", "MiniMax"),
|
| 541 |
+
("DeepSeek:", "DeepSeek"),
|
| 542 |
+
("OpenAI:", "OpenAI"),
|
| 543 |
+
("Google:", "Google"),
|
| 544 |
+
("Meta:", "Meta"),
|
| 545 |
+
("Qwen:", "Qwen"),
|
| 546 |
+
("Tencent:", "Tencent"),
|
| 547 |
+
("Z.ai:", "Z.ai"),
|
| 548 |
+
("NVIDIA:", "NVIDIA"),
|
| 549 |
+
("Cohere:", "Cohere"),
|
| 550 |
+
("Mistral:", "Mistral"),
|
| 551 |
+
)
|
| 552 |
+
for prefix, lab in prefixes:
|
| 553 |
+
if model_name.startswith(prefix):
|
| 554 |
+
return lab
|
| 555 |
+
return str(row.get("provider_name") or model_name.split(":", 1)[0])
|
| 556 |
+
|
| 557 |
+
|
| 558 |
+
def _model_label(model_name: str) -> str:
|
| 559 |
+
label = model_name.split(":", 1)[-1].strip()
|
| 560 |
+
return (
|
| 561 |
+
label.replace("GPT-5.6 ", "5.6 ")
|
| 562 |
+
.replace("Claude ", "")
|
| 563 |
+
.replace("DeepSeek ", "")
|
| 564 |
+
.replace("Command ", "")
|
| 565 |
+
)
|
| 566 |
+
|
| 567 |
+
|
| 568 |
+
_seen_labs: set[str] = set()
|
| 569 |
+
LAB_CHAMPIONS: list[dict[str, Any]] = []
|
| 570 |
+
for _model in DISPLAY_MODELS:
|
| 571 |
+
_lab = _lab_name(_model)
|
| 572 |
+
if _lab not in _seen_labs:
|
| 573 |
+
_seen_labs.add(_lab)
|
| 574 |
+
LAB_CHAMPIONS.append(_model)
|
| 575 |
|
| 576 |
|
| 577 |
def _completion_diagnostic(model_id: str) -> dict[str, Any]:
|
|
|
|
| 609 |
}
|
| 610 |
|
| 611 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 612 |
def _frontier_html() -> str:
|
| 613 |
+
axis_floor = 55.0
|
| 614 |
+
axis_ceiling = 66.0
|
| 615 |
rows = []
|
| 616 |
+
for place, model in enumerate(LAB_CHAMPIONS[:10], start=1):
|
| 617 |
score = float(model["flavourbench_score"])
|
| 618 |
+
position = max(0.0, min(100.0, (score - axis_floor) / (axis_ceiling - axis_floor) * 100))
|
| 619 |
+
full_name = str(model["model_name"])
|
| 620 |
+
lab = _lab_name(model)
|
| 621 |
+
label = _model_label(full_name)
|
| 622 |
rows.append(
|
| 623 |
"<div class='fb-forest-row'>"
|
| 624 |
+
f"<div class='fb-place'>{place:02d}</div>"
|
| 625 |
+
f"<div class='fb-model' title='{html.escape(full_name)}'>"
|
| 626 |
+
f"{html.escape(lab)} · {html.escape(label)}</div>"
|
| 627 |
"<div class='fb-axis'>"
|
| 628 |
+
f"<span class='fb-bar' style='width:{position:.3f}%'></span>"
|
| 629 |
+
f"<span class='fb-point' style='left:{position:.3f}%'></span>"
|
| 630 |
"</div>"
|
| 631 |
f"<div class='fb-number'>{score:.1f}</div>"
|
|
|
|
| 632 |
"</div>"
|
| 633 |
)
|
| 634 |
+
return "".join(rows) + (
|
| 635 |
+
f"<div class='fb-chart-foot'><span>{axis_floor:.0f}</span>"
|
| 636 |
+
f"<span>FlavourBench Score</span><span>{axis_ceiling:.0f}</span></div>"
|
| 637 |
+
)
|
| 638 |
|
| 639 |
|
| 640 |
def _hero_html() -> str:
|
| 641 |
inference = BUNDLE["analysis"]["inference"]
|
|
|
|
| 642 |
return f"""
|
| 643 |
<div class="fb-shell fb-hero">
|
| 644 |
<section>
|
| 645 |
+
<div class="fb-kicker">FlavourBench / executable benchmark</div>
|
| 646 |
+
<h1>Which AI knows food best?</h1>
|
| 647 |
+
<p class="fb-dek">Epicure scores every legal answer first. Then {MODEL_COUNT} frontier endpoints face the same {TASK_COUNT} food decisions.</p>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 648 |
<div class="fb-stats">
|
| 649 |
+
<div class="fb-stat"><strong>{MODEL_COUNT}</strong><span>frontier endpoints</span></div>
|
| 650 |
+
<div class="fb-stat"><strong>{TASK_COUNT}</strong><span>shared tasks</span></div>
|
| 651 |
+
<div class="fb-stat"><strong>{PRIMARY_COUNT:,}</strong><span>scored answers</span></div>
|
| 652 |
+
<div class="fb-stat"><strong>{inference["pairwise_hypotheses"]}</strong><span>model pairs</span></div>
|
| 653 |
</div>
|
| 654 |
</section>
|
| 655 |
+
<section class="fb-frontier" aria-label="Best FlavourBench model from each leading lab">
|
| 656 |
+
<div class="fb-frontier-head"><strong>Best model from each lab</strong><span>Point estimates · focused 55 to 66 axis</span></div>
|
| 657 |
{_frontier_html()}
|
| 658 |
</section>
|
| 659 |
</div>
|
|
|
|
| 668 |
rows.append(
|
| 669 |
{
|
| 670 |
"Rank": model["point_estimate_rank"],
|
|
|
|
| 671 |
"Model": model["model_name"],
|
| 672 |
+
"Score ↑": round(float(model["flavourbench_score"]), 2),
|
| 673 |
"Simultaneous 95%": f"{ci[0]:.2f} to {ci[1]:.2f}",
|
| 674 |
+
"Group": model["statistical_rank_group"],
|
| 675 |
"Rank 95%": f"{rank_ci[0]} to {rank_ci[1]}",
|
| 676 |
"Cells": f"{model['coverage']['valid_scored']}/{TASK_COUNT}",
|
|
|
|
| 677 |
}
|
| 678 |
)
|
| 679 |
return pd.DataFrame(rows)
|
| 680 |
|
| 681 |
|
| 682 |
+
def _leaderboard_html() -> str:
|
| 683 |
+
rows = []
|
| 684 |
+
for model in DISPLAY_MODELS:
|
| 685 |
+
ci = model["score_simultaneous_95_ci"]
|
| 686 |
+
rank_ci = model["bootstrap_rank_95_interval"]
|
| 687 |
+
rows.append(
|
| 688 |
+
"<tr>"
|
| 689 |
+
f"<td class='fb-rank-cell'>{int(model['point_estimate_rank']):02d}</td>"
|
| 690 |
+
f"<td>{html.escape(str(model['model_name']))}</td>"
|
| 691 |
+
f"<td class='fb-score-cell'>{float(model['flavourbench_score']):.2f}</td>"
|
| 692 |
+
f"<td>{float(ci[0]):.2f} to {float(ci[1]):.2f}</td>"
|
| 693 |
+
f"<td>G{model['statistical_rank_group']}</td>"
|
| 694 |
+
f"<td>{rank_ci[0]} to {rank_ci[1]}</td>"
|
| 695 |
+
f"<td>{model['coverage']['valid_scored']}/{TASK_COUNT}</td>"
|
| 696 |
+
"</tr>"
|
| 697 |
+
)
|
| 698 |
+
return """
|
| 699 |
+
<div class="fb-table-wrap">
|
| 700 |
+
<table class="fb-table">
|
| 701 |
+
<caption>Complete FlavourBench common-core leaderboard</caption>
|
| 702 |
+
<thead><tr>
|
| 703 |
+
<th scope="col">Rank</th><th scope="col">Model</th><th scope="col">Score</th>
|
| 704 |
+
<th scope="col">Simultaneous 95%</th><th scope="col">Group</th>
|
| 705 |
+
<th scope="col">Rank 95%</th><th scope="col">Cells</th>
|
| 706 |
+
</tr></thead>
|
| 707 |
+
<tbody>""" + "".join(rows) + """</tbody>
|
| 708 |
+
</table>
|
| 709 |
+
</div>
|
| 710 |
+
"""
|
| 711 |
+
|
| 712 |
+
|
| 713 |
def _model_detail(model_name: str) -> tuple[str, pd.DataFrame]:
|
| 714 |
model = MODEL_BY_NAME[model_name]
|
| 715 |
rank_interval = model["bootstrap_rank_95_interval"]
|
|
|
|
| 865 |
}
|
| 866 |
|
| 867 |
|
| 868 |
+
def _command_preview(
|
| 869 |
+
runtime: str,
|
| 870 |
+
model: str,
|
| 871 |
+
base_url: str,
|
| 872 |
+
api_key_env: str,
|
| 873 |
+
scope: str,
|
| 874 |
+
) -> str:
|
| 875 |
+
"""Render a copyable lab command without receiving model credentials."""
|
| 876 |
+
|
| 877 |
+
model = " ".join(str(model or "").split()) or "your-exact-model-id"
|
| 878 |
+
base_url = " ".join(str(base_url or "").split()) or "https://your-endpoint.example/v1"
|
| 879 |
+
api_key_env = "".join(character for character in str(api_key_env or "") if character.isalnum() or character == "_")
|
| 880 |
+
api_key_env = api_key_env or "LAB_MODEL_API_KEY"
|
| 881 |
+
smoke = " --limit 12" if scope.startswith("12-task") else ""
|
| 882 |
+
backend = "transformers" if runtime.startswith("Local") else "openai-compatible"
|
| 883 |
+
route = ""
|
| 884 |
+
if backend == "openai-compatible":
|
| 885 |
+
route = f" \\\n+ --base-url {shlex.quote(base_url)} \\\n+ --api-key-env {shlex.quote(api_key_env)}"
|
| 886 |
+
command = (
|
| 887 |
+
"python -m pip install \"epicure-flavourbench @ "
|
| 888 |
+
"git+https://github.com/josefchen/flavourbench.git\"\n\n"
|
| 889 |
+
f"flavourbench run \\\n+ --backend {backend} \\\n+ --model {shlex.quote(model)}"
|
| 890 |
+
f"{route} \\\n+ --responses responses.jsonl \\\n+ --report flavourbench-report.json \\\n+ --resume{smoke}"
|
| 891 |
+
)
|
| 892 |
+
note = (
|
| 893 |
+
f"Set `{api_key_env}` in your shell before running. The key stays on your machine."
|
| 894 |
+
if backend == "openai-compatible"
|
| 895 |
+
else "The checkpoint runs locally through Transformers. No endpoint credential is used."
|
| 896 |
+
)
|
| 897 |
+
return f"```bash\n{command}\n```\n\n{note}"
|
| 898 |
+
|
| 899 |
+
|
| 900 |
+
def _reward_preview(task_label: str, completion: str) -> str:
|
| 901 |
+
"""Score one development completion for the interactive reward demonstration."""
|
| 902 |
+
|
| 903 |
+
task_id = LAB_TASK_LABEL_TO_ID[task_label]
|
| 904 |
+
try:
|
| 905 |
+
result = _training_reward_api(task_id, completion)
|
| 906 |
+
except SpaceLabError as error:
|
| 907 |
+
return f"<div class='fb-evidence'><strong>Not parseable.</strong> {html.escape(str(error))}</div>"
|
| 908 |
+
selection = html.escape(str(result.get("selection") or result.get("observed_selection") or ""))
|
| 909 |
+
return f"""
|
| 910 |
+
<div class="fb-evidence">
|
| 911 |
+
<strong>Reward {float(result['reward']):.4f}</strong> · score {float(result['score']):.2f} · selection <code>{selection}</code><br>
|
| 912 |
+
This map belongs to the anchor-disjoint development track. It cannot alter the public leaderboard.
|
| 913 |
+
</div>
|
| 914 |
+
"""
|
| 915 |
+
|
| 916 |
+
|
| 917 |
def _score_upload(
|
| 918 |
artifact_path: str | None,
|
| 919 |
model_name: str,
|
|
|
|
| 989 |
|
| 990 |
theme = gr.themes.Base(
|
| 991 |
primary_hue=gr.themes.Color(
|
| 992 |
+
c50="#F8ECE9",
|
| 993 |
+
c100="#F1DDD8",
|
| 994 |
+
c200="#E7BCB5",
|
| 995 |
+
c300="#D98F84",
|
| 996 |
+
c400="#C96357",
|
| 997 |
+
c500=RUST,
|
| 998 |
+
c600="#98352D",
|
| 999 |
+
c700="#7C2E28",
|
| 1000 |
+
c800="#662A25",
|
| 1001 |
+
c900="#562722",
|
| 1002 |
+
c950="#2F120F",
|
| 1003 |
),
|
| 1004 |
+
neutral_hue="stone",
|
| 1005 |
font=gr.themes.GoogleFont("Geist", weights=(400, 500, 600, 700)),
|
| 1006 |
font_mono=gr.themes.GoogleFont("IBM Plex Mono", weights=(400, 600)),
|
| 1007 |
).set(
|
| 1008 |
+
body_background_fill="#F4F3EE",
|
| 1009 |
+
block_background_fill="#F4F3EE",
|
| 1010 |
+
block_border_width="0px",
|
| 1011 |
block_label_text_weight="600",
|
| 1012 |
+
button_primary_background_fill=INK,
|
| 1013 |
+
button_primary_background_fill_hover=RUST,
|
| 1014 |
)
|
| 1015 |
|
| 1016 |
|
| 1017 |
+
with gr.Blocks(title="FlavourBench | Which AI knows food best?") as demo:
|
| 1018 |
gr.HTML(_hero_html())
|
| 1019 |
with gr.Tabs():
|
| 1020 |
+
with gr.Tab("Leaders"):
|
| 1021 |
gr.HTML(
|
| 1022 |
+
f"""
|
| 1023 |
<div class="fb-section">
|
| 1024 |
+
<h2>The full scoreboard</h2>
|
| 1025 |
+
<p>One score for the same {TASK_COUNT} tasks. Intervals and rank groups show which gaps the shared-task evidence can actually resolve.</p>
|
| 1026 |
</div>
|
| 1027 |
"""
|
| 1028 |
)
|
| 1029 |
+
gr.HTML(_leaderboard_html())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1030 |
gr.Markdown(
|
| 1031 |
+
"**Read the score first.** The focused chart above uses point estimates. "
|
| 1032 |
+
"The table adds simultaneous 95% bands, bootstrap rank intervals, and multiplicity-controlled groups. "
|
| 1033 |
+
"A score of 100 means selecting Epicure's optimum on every task."
|
| 1034 |
)
|
| 1035 |
|
| 1036 |
+
with gr.Tab("Profiles"):
|
| 1037 |
gr.HTML(
|
| 1038 |
"""
|
| 1039 |
<div class="fb-section">
|
| 1040 |
+
<h2>Where each model wins</h2>
|
| 1041 |
+
<p>Break the headline score into substitution, pairing, and constraint performance, then compare the two collection panels.</p>
|
| 1042 |
</div>
|
| 1043 |
"""
|
| 1044 |
)
|
|
|
|
| 1059 |
_model_detail,
|
| 1060 |
inputs=model_selector,
|
| 1061 |
outputs=[model_summary, family_table],
|
| 1062 |
+
api_visibility="private",
|
| 1063 |
)
|
| 1064 |
|
| 1065 |
+
with gr.Tab("Inspect"):
|
| 1066 |
gr.HTML(
|
| 1067 |
"""
|
| 1068 |
<div class="fb-section">
|
| 1069 |
<h2>Open one scored decision</h2>
|
| 1070 |
+
<p>Every answer is traceable to the exact prompt, model response, and precomputed 56-choice reward surface.</p>
|
| 1071 |
</div>
|
| 1072 |
"""
|
| 1073 |
)
|
|
|
|
| 1086 |
filterable=True,
|
| 1087 |
scale=2,
|
| 1088 |
)
|
| 1089 |
+
inspect_task = gr.Button("Inspect task", variant="primary", scale=0)
|
| 1090 |
initial = _task_detail(MODEL_NAMES[0], next(iter(TASK_LABEL_TO_ID)))
|
| 1091 |
task_status = gr.HTML(initial[0])
|
| 1092 |
+
prompt = gr.Textbox(value=initial[1], label="Exact prompt", lines=11, interactive=False)
|
| 1093 |
+
choices = gr.JSON(value=initial[2], label="Eight candidates")
|
| 1094 |
score_map = gr.Dataframe(
|
| 1095 |
value=initial[3],
|
| 1096 |
+
label="Top 12 of 56 scored selections",
|
| 1097 |
interactive=False,
|
| 1098 |
wrap=True,
|
| 1099 |
show_row_numbers=False,
|
| 1100 |
)
|
| 1101 |
answer = gr.Markdown(value=initial[4], label="Model response")
|
| 1102 |
+
provenance = gr.Markdown(value=initial[5], label="Content hashes and route")
|
| 1103 |
inspect_task.click(
|
| 1104 |
_task_detail,
|
| 1105 |
inputs=[task_model, task_selector],
|
| 1106 |
outputs=[task_status, prompt, choices, score_map, answer, provenance],
|
| 1107 |
+
api_visibility="private",
|
| 1108 |
)
|
| 1109 |
|
| 1110 |
+
with gr.Tab("Run your model"):
|
| 1111 |
gr.HTML(
|
| 1112 |
f"""
|
| 1113 |
<div class="fb-section">
|
| 1114 |
+
<h2>Run FlavourBench on any model</h2>
|
| 1115 |
+
<p>Your endpoint key or checkpoint stays in your environment. The open runner fetches the exact task set, resumes interrupted jobs, and writes a verifiable report.</p>
|
| 1116 |
+
</div>
|
| 1117 |
+
<div class="fb-step-line">
|
| 1118 |
+
<div class="fb-step"><b>01 / Connect</b><strong>Endpoint or checkpoint</strong><span>OpenAI-compatible APIs and local Transformers models use the same task contract.</span></div>
|
| 1119 |
+
<div class="fb-step"><b>02 / Run</b><strong>{TASK_COUNT} shared decisions</strong><span>Start with 12 tasks, then resume into the complete run without repeating finished calls.</span></div>
|
| 1120 |
+
<div class="fb-step"><b>03 / Verify</b><strong>Score plus uncertainty</strong><span>The report binds the task set, responses, score, coverage, and inference by hash.</span></div>
|
| 1121 |
+
</div>
|
| 1122 |
+
"""
|
| 1123 |
+
)
|
| 1124 |
+
with gr.Row():
|
| 1125 |
+
with gr.Column(scale=1):
|
| 1126 |
+
runtime = gr.Radio(
|
| 1127 |
+
choices=["Hosted OpenAI-compatible endpoint", "Local Transformers checkpoint"],
|
| 1128 |
+
value="Hosted OpenAI-compatible endpoint",
|
| 1129 |
+
label="Runtime",
|
| 1130 |
+
)
|
| 1131 |
+
run_model = gr.Textbox(
|
| 1132 |
+
value="your-exact-model-id",
|
| 1133 |
+
label="Model ID or checkpoint",
|
| 1134 |
+
)
|
| 1135 |
+
run_scope = gr.Radio(
|
| 1136 |
+
choices=["12-task smoke test", f"Full {TASK_COUNT}-task benchmark"],
|
| 1137 |
+
value="12-task smoke test",
|
| 1138 |
+
label="Run size",
|
| 1139 |
+
)
|
| 1140 |
+
with gr.Column(scale=1):
|
| 1141 |
+
run_base_url = gr.Textbox(
|
| 1142 |
+
value="https://your-endpoint.example/v1",
|
| 1143 |
+
label="Base URL for hosted endpoints",
|
| 1144 |
+
)
|
| 1145 |
+
run_key_env = gr.Textbox(
|
| 1146 |
+
value="LAB_MODEL_API_KEY",
|
| 1147 |
+
label="Local environment variable containing the key",
|
| 1148 |
+
)
|
| 1149 |
+
generate_command = gr.Button("Build command", variant="primary")
|
| 1150 |
+
initial_command = _command_preview(
|
| 1151 |
+
"Hosted OpenAI-compatible endpoint",
|
| 1152 |
+
"your-exact-model-id",
|
| 1153 |
+
"https://your-endpoint.example/v1",
|
| 1154 |
+
"LAB_MODEL_API_KEY",
|
| 1155 |
+
"12-task smoke test",
|
| 1156 |
+
)
|
| 1157 |
+
run_command = gr.Markdown(initial_command)
|
| 1158 |
+
generate_command.click(
|
| 1159 |
+
_command_preview,
|
| 1160 |
+
inputs=[runtime, run_model, run_base_url, run_key_env, run_scope],
|
| 1161 |
+
outputs=run_command,
|
| 1162 |
+
api_visibility="private",
|
| 1163 |
+
)
|
| 1164 |
+
|
| 1165 |
+
gr.HTML(
|
| 1166 |
+
"""
|
| 1167 |
+
<div class="fb-section">
|
| 1168 |
+
<h2>Try the training reward</h2>
|
| 1169 |
+
<p>Paste one completion and query an anchor-disjoint development map. This is the same deterministic reward used by the local GRPO recipe.</p>
|
| 1170 |
</div>
|
| 1171 |
+
"""
|
| 1172 |
+
)
|
| 1173 |
+
initial_lab_label = next(iter(LAB_TASK_LABEL_TO_ID))
|
| 1174 |
+
with gr.Row():
|
| 1175 |
+
reward_task = gr.Dropdown(
|
| 1176 |
+
choices=list(LAB_TASK_LABEL_TO_ID),
|
| 1177 |
+
value=initial_lab_label,
|
| 1178 |
+
label="Development task",
|
| 1179 |
+
filterable=True,
|
| 1180 |
+
scale=2,
|
| 1181 |
+
)
|
| 1182 |
+
reward_completion = gr.Textbox(
|
| 1183 |
+
value="FINAL_SELECTION: A,B,C",
|
| 1184 |
+
label="Model completion",
|
| 1185 |
+
scale=2,
|
| 1186 |
+
)
|
| 1187 |
+
score_reward = gr.Button("Score answer", variant="primary", scale=0)
|
| 1188 |
+
reward_result = gr.HTML(_reward_preview(initial_lab_label, "FINAL_SELECTION: A,B,C"))
|
| 1189 |
+
score_reward.click(
|
| 1190 |
+
_reward_preview,
|
| 1191 |
+
inputs=[reward_task, reward_completion],
|
| 1192 |
+
outputs=reward_result,
|
| 1193 |
+
api_visibility="private",
|
| 1194 |
+
)
|
| 1195 |
+
|
| 1196 |
+
gr.HTML(
|
| 1197 |
+
f"""
|
| 1198 |
+
<div class="fb-section">
|
| 1199 |
+
<h2>Score a completed run</h2>
|
| 1200 |
+
<p>Upload one JSON or JSONL response per task. Complete runs receive a FlavourBench Score; partial runs receive diagnostics only.</p>
|
| 1201 |
</div>
|
| 1202 |
+
<div class="fb-evidence"><strong>Comparable means complete.</strong> All {TASK_COUNT} responses must be present and parseable. Uploads are scored in-session and never added to the public leaderboard automatically.</div>
|
| 1203 |
"""
|
| 1204 |
)
|
| 1205 |
with gr.Row():
|
|
|
|
| 1224 |
file_types=[".jsonl", ".json"],
|
| 1225 |
type="filepath",
|
| 1226 |
)
|
| 1227 |
+
score_upload = gr.Button("Score complete artifact", variant="primary")
|
| 1228 |
lab_summary = gr.Markdown()
|
| 1229 |
lab_rows = gr.Dataframe(
|
| 1230 |
interactive=False,
|
|
|
|
| 1242 |
)
|
| 1243 |
gr.Markdown(
|
| 1244 |
"""
|
|
|
|
|
|
|
| 1245 |
```json
|
| 1246 |
{"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}
|
| 1247 |
```
|
| 1248 |
|
| 1249 |
+
The named `score_completion`, `score_submission`, and `training_reward` endpoints appear under
|
| 1250 |
+
**Use via API**. For high-throughput RL, use the local reward lookup and the runnable SFT, DPO,
|
| 1251 |
+
and GRPO recipes in the source repository.
|
|
|
|
|
|
|
| 1252 |
"""
|
| 1253 |
)
|
| 1254 |
|
| 1255 |
+
with gr.Tab("Compare"):
|
| 1256 |
gr.HTML(
|
| 1257 |
f"""
|
| 1258 |
<div class="fb-section">
|
| 1259 |
+
<h2>Does the gap hold up?</h2>
|
| 1260 |
+
<p>Query any of the {PAIR_COUNT} paired model contrasts on the same tasks, with Holm control across the full comparison family.</p>
|
| 1261 |
</div>
|
| 1262 |
"""
|
| 1263 |
)
|
|
|
|
| 1274 |
label="Second model",
|
| 1275 |
filterable=True,
|
| 1276 |
)
|
| 1277 |
+
compare = gr.Button("Compare models", variant="primary", scale=0)
|
| 1278 |
pair_result = gr.HTML(_pair_detail(MODEL_NAMES[0], MODEL_NAMES[1]))
|
| 1279 |
+
compare.click(
|
| 1280 |
+
_pair_detail,
|
| 1281 |
+
inputs=[left_model, right_model],
|
| 1282 |
+
outputs=pair_result,
|
| 1283 |
+
api_visibility="private",
|
| 1284 |
+
)
|
| 1285 |
|
| 1286 |
+
with gr.Tab("Method"):
|
| 1287 |
gr.HTML(
|
| 1288 |
f"""
|
| 1289 |
<div class="fb-section">
|
| 1290 |
+
<h2>One lookup, repeated {TASK_COUNT} times</h2>
|
| 1291 |
+
<p>The Space makes no provider calls. It reads released reward maps and returns deterministic scores.</p>
|
| 1292 |
+
</div>
|
| 1293 |
+
<div class="fb-step-line">
|
| 1294 |
+
<div class="fb-step"><b>Task</b><strong>Pick 3 of 8</strong><span>Each prompt defines exactly 56 legal portfolios.</span></div>
|
| 1295 |
+
<div class="fb-step"><b>Ground truth</b><strong>Score all 56 first</strong><span>Epicure constructs the task map before any evaluated model runs.</span></div>
|
| 1296 |
+
<div class="fb-step"><b>Benchmark</b><strong>Aggregate 534 choices</strong><span>The metric equal-weights substitution, pairing, and constraint families.</span></div>
|
| 1297 |
</div>
|
| 1298 |
<div class="fb-method">
|
| 1299 |
<div>
|
| 1300 |
<h3>Scoring contract</h3>
|
| 1301 |
+
<p>A task score ranges from 0 to 100 on its released Epicure map. The complete
|
| 1302 |
+
release is a {MODEL_COUNT} by {TASK_COUNT} matrix with one valid response in every cell.</p>
|
|
|
|
|
|
|
| 1303 |
<h3>Inference</h3>
|
| 1304 |
<p>Results use {INDEPENDENT_CLUSTER_COUNT:,} ingredient-anchor clusters,
|
| 1305 |
+
50,000 shared cluster bootstraps, simultaneous score bands, 100,000 cluster
|
| 1306 |
+
sign flips, Holm correction, exact tests against a random legal choice, bootstrap rank intervals,
|
| 1307 |
+
and an independently compiled second panel.</p>
|
| 1308 |
+
<h3>Training boundary</h3>
|
| 1309 |
+
<p>The 426 SFT, DPO, and GRPO development maps use anchors that do not occur in
|
| 1310 |
+
the {TASK_COUNT}-task leaderboard. Training cannot query the official test reward maps by accident.</p>
|
| 1311 |
</div>
|
| 1312 |
<aside class="fb-evidence">
|
| 1313 |
+
<strong>Content-addressed release</strong><br>
|
| 1314 |
<span class="fb-hash">{BUNDLE["release_artifact_sha256"]}</span><br><br>
|
| 1315 |
+
{MODEL_COUNT} endpoints<br>{TASK_COUNT} tasks<br>{INDEPENDENT_CLUSTER_COUNT:,} anchor clusters<br>{PRIMARY_COUNT:,} complete answers<br>{BUNDLE["analysis"]["resolved_pair_count"]}/{PAIR_COUNT} resolved pairs
|
| 1316 |
</aside>
|
| 1317 |
</div>
|
| 1318 |
"""
|
|
|
|
| 1322 |
```bash
|
| 1323 |
git clone https://github.com/josefchen/flavourbench.git
|
| 1324 |
cd flavourbench
|
| 1325 |
+
python -m pip install -e '.[dev]'
|
| 1326 |
+
pytest -q tests/lab_cli_test.py tests/hf_lab_space_api_test.py
|
|
|
|
| 1327 |
```
|
|
|
|
|
|
|
| 1328 |
"""
|
| 1329 |
)
|
| 1330 |
|
|
|
|
| 1350 |
gr.HTML(
|
| 1351 |
"""
|
| 1352 |
<div class="fb-shell fb-footer">
|
| 1353 |
+
<div>FlavourBench · Josef Chen, Independent Researcher · Erim Hayretci, Imperial College London</div>
|
| 1354 |
+
<span><a href="https://github.com/josefchen/flavourbench/blob/main/paper/build/flavourbench.pdf">Paper</a> · <a href="https://huggingface.co/datasets/josefchen/flavourbench">Dataset</a> · <a href="https://github.com/josefchen/flavourbench">Source</a></span>
|
| 1355 |
</div>
|
| 1356 |
"""
|
| 1357 |
)
|
assets/flavourbench-leaderboard.svg
ADDED
|
|