README.md

metadata

title: FlavourBench
emoji: 🍲
colorFrom: red
colorTo: red
sdk: gradio
sdk_version: 6.9.0
app_file: app.py
pinned: false
license: other
datasets:
  - josefchen/flavourbench
tags:
  - leaderboard
  - modality:text
  - judge:auto
  - submission:manual
  - test:public
  - reproducibility
  - culinary

FlavourBench

Pick 3 ingredients from 8. Epicure scores all 56 legal portfolios first. Then every model faces the same 534 decisions.

The Space is both the public scorebook and a working benchmark interface:

• Leaders ranks all 27 endpoints with simultaneous intervals and statistical groups.
• Insights shows score bands, pairwise resolution, and task-family fingerprints.
• Profiles breaks each score into substitution, pairing, and constraint performance.
• Inspect opens the exact prompt, answer, 56-choice score map, route, and content hashes.
• Run your model builds a copyable endpoint or local-checkpoint command, demonstrates one dense training reward, and scores a complete JSONL run.
• Compare queries any of the 351 shared-task pairwise contrasts.

No model judge runs behind the interface. The Space performs deterministic lookups against the released reward maps and makes no model-provider calls.

Run from your own environment

python -m pip install "epicure-flavourbench @ git+https://github.com/josefchen/flavourbench.git"

export LAB_MODEL_API_KEY='...'
flavourbench run \
  --backend openai-compatible \
  --base-url https://your-endpoint.example/v1 \
  --api-key-env LAB_MODEL_API_KEY \
  --model your-exact-model-id \
  --responses responses.jsonl \
  --report flavourbench-report.json \
  --resume

Add --limit 12 for a balanced smoke test. The runner checkpoints each answer and resumes without repeating completed calls. Credentials and model weights stay in your environment.

The accepted response contract is one JSON object per line:

{"task_id":"...","status":"completed","response":"FINAL_SELECTION: A,B,C"}

A comparable score requires one valid answer for all 534 tasks. Partial runs still receive per-task and coverage diagnostics. Uploads are never added to the official leaderboard automatically.

Verified complete runs can be proposed through the result submission form. The submission contract lists the required response artifact, route metadata, settings, and training disclosure. Accepted results enter a new versioned release; the published release is never edited in place.

API and training

The Space exposes four named endpoints:

Endpoint	Use
/score_completion	Score one completion on one official task
/score_submission	Score a complete JSON or JSONL artifact supplied as text
/training_reward	Query one of 342 anchor-disjoint train/validation reward maps
/score_uploaded_submission	Score an uploaded artifact and return a report

Use Use via API in the running Space for generated Python, JavaScript, and curl clients. For high-throughput RL, use the local deterministic reward function. The linked dataset includes ready-to-load SFT, DPO, and GRPO views plus runnable LoRA recipes for Hugging Face Jobs.

The prospective reward-transfer protocol defines the 84-task transfer split, seeds, six confirmatory contrasts, and multiplicity control.

Dataset and lab kit    Paper    Source

Josef Chen, Independent Researcher
Erim Hayretci, Imperial College London

@article{chen2026flavourbench,
  title  = {FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth},
  author = {Chen, Josef and Hayretci, Erim},
  journal = {arXiv preprint arXiv:2608.20574},
  year   = {2026},
  eprint = {2608.20574},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2608.20574}
}