Spaces:
Running
Running
| # SearchGen-Bench Per-Prompt Data and Leaderboard Design | |
| ## Implementation status | |
| This design is implemented in the Space: | |
| - `scripts/build_leaderboard_data.py` imports the canonical ToolGen extraction | |
| function, exports all prompt-model records, validates them, and generates the | |
| four aggregate views. | |
| - `scripts/validate_leaderboard_data.py` independently reconstructs all | |
| committed aggregates from `prompt_scores.jsonl` without requiring ToolGen. | |
| - `public/data/manifest.json` pins the schema, input hash, components, model | |
| source IDs, partitions, and missing-result policies. | |
| - The frontend provides component/stratum rankings plus domain and failure-mode | |
| scoreboards. | |
| - Hugging Face serves the locally built `dist/` contents published at the Space | |
| repository root and performs no build or mutable runtime data download. A | |
| Dockerfile remains available for optional container deployment. | |
| To regenerate a release: | |
| ```bash | |
| TOOLGEN_ROOT=/path/to/ToolGen npm run build:data | |
| npm run test:data | |
| npm run build | |
| ``` | |
| ## 1. Decision | |
| The proposed design is sound: create one versioned record for each of the 751 | |
| evaluation prompts, attach its benchmark tags, and record per-model component | |
| scores. All public scoreboards should be derived from this data rather than | |
| maintained as hand-written tables. | |
| One metric distinction must remain explicit: | |
| - The paper's canonical score is the mean of 10 applicable components. | |
| - The proposed public leaderboard displays nine components by excluding | |
| `text_reference_evaluation` (called "text fidelity" in the proposal). | |
| - `text_rendering` remains included; it is not the excluded component. | |
| - A mean recomputed from the remaining nine components must be named | |
| **Overall-9**. It is not numerically interchangeable with the paper's | |
| canonical **Overall-10**. | |
| Do not discard the tenth component at extraction time. Preserve all 10 raw | |
| components so the exported data can reproduce the paper, then define the | |
| nine-component leaderboard as a named metric view. This avoids an irreversible | |
| data loss and makes discrepancies auditable. | |
| ## 2. Canonical inputs | |
| Use the sources defined by `TABLE_UPDATE_SOP.md`: | |
| - Prompt metadata: | |
| `phase4d1_generations/eval_datasets_548_finalized_v1.jsonl` | |
| - Evaluation result: | |
| `final_20k_release_v2/{release_row}/{lane}/{generator}/augmented_parsed_result_ffjudge_pp.json` | |
| - Extraction semantics: reuse `extract_10comp()` from | |
| `paper_materials/recompute_tables.py`; do not create a second implementation | |
| of the component rules. | |
| The markdown SOP is documentation, not a machine-readable input. The JSONL and | |
| evaluation result files remain the ground truth. | |
| ## 3. Prompt identity and tags | |
| Use `sample_id` as the stable primary key. An integer `prompt_index` may be | |
| included for presentation, but it must never be used to join data because file | |
| ordering can change. | |
| Each prompt record should contain: | |
| - `sample_id` | |
| - `prompt_index` | |
| - `release_row` | |
| - `stratum`: `NoSearch` or `SearchIntensive` | |
| - `search_type`: `NoSearch`, `VisualSearch`, or `TextualSearch` | |
| - `domains`: list of zero or more domain tags | |
| - `failure_modes`: list of zero or more failure-mode tags | |
| - Optional useful filters: `difficulty`, `language`, | |
| `generation_task_type`, and `is_miniset` | |
| Derive the partitions exactly as specified by the SOP: | |
| ```python | |
| if row.get("subset") == "NoSearch": | |
| stratum = "NoSearch" | |
| search_type = "NoSearch" | |
| elif "texthard" in row["sample_id"] or "text_rendering" in row["sample_id"]: | |
| stratum = "SearchIntensive" | |
| search_type = "TextualSearch" | |
| else: | |
| stratum = "SearchIntensive" | |
| search_type = "VisualSearch" | |
| ``` | |
| Expected counts are: | |
| | Partition | Count | | |
| |---|---:| | |
| | NoSearch | 100 | | |
| | SearchIntensive | 651 | | |
| | VisualSearch | 387 | | |
| | TextualSearch | 264 | | |
| | Total | 751 | | |
| ## 4. Score schema | |
| Use a normalized, lossless JSONL artifact for the canonical public data. One | |
| prompt record can contain a `models` map to avoid repeating prompt metadata: | |
| ```json | |
| { | |
| "sample_id": "eval_base40_001", | |
| "prompt_index": 0, | |
| "release_row": "evalset_rows/eval_base40_001", | |
| "stratum": "SearchIntensive", | |
| "search_type": "VisualSearch", | |
| "domains": ["CUL", "ART", "TEC", "ARC"], | |
| "failure_modes": ["TK-R", "EIK", "CS", "THG", "CCS"], | |
| "models": { | |
| "gpt_image": { | |
| "status": "scored", | |
| "lane": "none", | |
| "components_raw_0to3": { | |
| "checklist": 2.1, | |
| "rubric_adaptive": 2.0, | |
| "prompt_faithfulness": 2.2, | |
| "image_quality": 2.4, | |
| "text_rendering": 2.1, | |
| "ai_naturalness": 2.0, | |
| "composition_and_aesthetics": 2.5, | |
| "physical_plausibility": 2.3, | |
| "visual_reference_evaluation": 1.9, | |
| "text_reference_evaluation": null | |
| }, | |
| "overall_10_raw": 2.1667, | |
| "overall_9_raw": 2.1667 | |
| } | |
| } | |
| } | |
| ``` | |
| Store unrounded 0–3 values, or values with enough precision to reproduce the | |
| aggregates. Convert to 0–100 and round only in derived artifacts or at display | |
| time. | |
| `null` means not applicable. It is different from a missing result and from a | |
| real score of zero. Represent result state separately: | |
| - `scored`: evaluation exists and was parsed | |
| - `missing_generation`: no generated image/result | |
| - `missing_evaluation`: image may exist but evaluation is absent | |
| - `invalid_evaluation`: file exists but is malformed or unusable | |
| ## 5. Metric definitions | |
| The nine displayed components are: | |
| 1. Checklist | |
| 2. Rubric Adaptive | |
| 3. Prompt Faithfulness | |
| 4. Image Quality | |
| 5. Text Rendering | |
| 6. AI Naturalness | |
| 7. Composition and Aesthetics | |
| 8. Physical Plausibility | |
| 9. Visual Reference | |
| The excluded public component is Text Reference: | |
| `text_reference_evaluation`. | |
| For a scored prompt-model pair: | |
| ```text | |
| Overall-10 = mean(non-null values among all 10 components) | |
| Overall-9 = mean(non-null values among the nine displayed components) | |
| Public scale = raw score * 100 / 3 | |
| ``` | |
| For a group such as a domain or failure mode: | |
| ```text | |
| Group Overall-9 = mean(per-prompt Overall-9 values included by policy) | |
| Group component = mean(non-null per-prompt values for that component) | |
| ``` | |
| This is prompt-macro averaging. Do not pool checklist items or rubric items | |
| across prompts, because prompts with more items would receive more weight. | |
| ## 6. Missing-result and coverage policy | |
| Raw data must preserve missing states. The aggregation layer applies the paper's | |
| model-specific policy: | |
| - Default: a missing result is zero-filled for aggregate computation. | |
| - `xai_image` and `qwen_image_2_pro`: missing generation/evaluation results are | |
| excluded from the denominator. | |
| - Inapplicable components remain `null` and are excluded from the relevant | |
| component denominator. | |
| Every aggregate row must publish: | |
| - `n_total`: prompts in the selected group | |
| - `n_scored`: prompts with valid evaluations | |
| - `n_included`: prompts included after applying the missing-result policy | |
| - `coverage`: `n_scored / n_total` | |
| - `missing_policy`: `zero_fill` or `exclude` | |
| Rankings should either require a declared minimum coverage or visibly flag low | |
| coverage. A high score with materially incomplete coverage must not appear | |
| equivalent to a fully covered score. | |
| ## 7. Multi-label breakdown semantics | |
| `domains` and `failure_modes` are multi-label fields. A prompt contributes once | |
| to every tag assigned to it. Therefore: | |
| - Domain row counts overlap and must not be summed to obtain 751. | |
| - Failure-mode row counts also overlap. | |
| - The UI must label these as multi-label breakdowns. | |
| - Each breakdown row must show its own prompt count and model coverage. | |
| Avoid fractional weighting across tags unless a separate, explicitly named | |
| analysis requires it. Full membership is easier to explain and audit. | |
| ## 8. Derived scoreboards | |
| Generate the following views from the same prompt-level artifact. | |
| ### 8.1 Overall model leaderboard | |
| One row per model, ranked by Overall-9. Show: | |
| - Rank and model metadata | |
| - Overall-9 | |
| - Overall-10, labeled as the paper-compatible score | |
| - All nine public components | |
| - Coverage and missing policy | |
| ### 8.2 Stratum leaderboard | |
| Provide filters or tabs for: | |
| - All 751 | |
| - NoSearch (100) | |
| - SearchIntensive (651) | |
| - VisualSearch (387) | |
| - TextualSearch (264) | |
| Each view shows Overall-9, the nine components, and coverage. | |
| ### 8.3 Domain breakdown | |
| For a selected model, show each domain's Overall-9, prompt count, and coverage. | |
| Optionally allow expanding a domain to reveal its nine components. This is more | |
| readable than placing every domain-component combination in one wide table. | |
| ### 8.4 Failure-mode breakdown | |
| Use the same layout and aggregation rules as the domain breakdown. | |
| ### 8.5 Comparison view | |
| Allow users to select a small number of models and compare their component | |
| profiles within a selected stratum, domain, or failure mode. This should use the | |
| same aggregate file rather than recomputing with different rules. | |
| ## 9. Artifact layout | |
| Recommended generated files: | |
| ```text | |
| public/data/ | |
| ├── manifest.json | |
| ├── prompt_scores.jsonl | |
| ├── leaderboard_overall.json | |
| ├── leaderboard_by_stratum.json | |
| ├── leaderboard_by_domain.json | |
| └── leaderboard_by_failure_mode.json | |
| ``` | |
| `manifest.json` should include: | |
| - Schema version | |
| - Benchmark dataset version/hash | |
| - Scoring version, including `ffjudge_pp` and PP configuration | |
| - Generation timestamp | |
| - Source repository commit, when available | |
| - Component lists for Overall-10 and Overall-9 | |
| - Scale and rounding rules | |
| - Missing-result policies | |
| - Model display names, types, and ordering metadata | |
| The browser should normally load the compact aggregate JSON files. Ship | |
| `prompt_scores.jsonl` for audit/download or client-side custom exploration, not | |
| as a prerequisite for rendering the initial table. | |
| ## 10. Implementation actions | |
| ### Phase A: Freeze semantics | |
| 1. Confirm that "drop text fidelity" means excluding | |
| `text_reference_evaluation`, not `text_rendering`. | |
| 2. Approve `Overall-9` as the public primary metric and retain `Overall-10` for | |
| paper compatibility. | |
| 3. Freeze model IDs, display names, model types, lane selection, missing-result | |
| policies, and minimum ranking coverage. | |
| ### Phase B: Build the exporter | |
| 1. Add a producer-side script in ToolGen, for example | |
| `paper_materials/export_searchgen_leaderboard.py`. | |
| 2. Import or refactor and reuse `extract_10comp()` rather than copying it. | |
| 3. Load all 751 prompt metadata rows and assert unique `sample_id` values. | |
| 4. Derive the two-level partition and copy all list-valued tags. | |
| 5. Resolve each configured model's canonical `none`-lane PP evaluation. | |
| 6. Record all 10 component values plus explicit result status. | |
| 7. Compute per-prompt Overall-10 and Overall-9. | |
| 8. Write the prompt JSONL and manifest deterministically. | |
| The exporter belongs near the canonical ToolGen data. The Hugging Face Space | |
| must not attempt to traverse a local ToolGen path at runtime. | |
| ### Phase C: Build and validate aggregates | |
| 1. Add a second script that reads only the exported JSONL and manifest. | |
| 2. Apply the declared missing policies and macro-aggregation rules. | |
| 3. Produce overall, stratum, domain, and failure-mode JSON files. | |
| 4. Sort output keys/rows deterministically to make reviews meaningful. | |
| 5. Compare Overall-10 outputs against `recompute_tables.py --json` before | |
| accepting a release. | |
| 6. Snapshot expected Overall-9 results in tests. | |
| ### Phase D: Replace the inherited BrowseComp data path | |
| 1. Remove the download of `Tevatron/BrowseComp-Plus-results` from | |
| `scripts/extract_data.py`. | |
| 2. Remove the runtime CSV-to-JSON extraction from the Docker startup command. | |
| 3. Generate and commit/copy the versioned SearchGen artifacts into | |
| `public/data/` before the Vite build, or download one pinned artifact release | |
| from a dedicated SearchGen dataset repository. | |
| 4. Fail the build if data validation fails; do not deploy an empty leaderboard. | |
| ### Phase E: Update the frontend | |
| 1. Replace the BrowseComp-specific columns in `main.js`. | |
| 2. Add table selectors for overall, strata, domains, and failure modes. | |
| 3. Make Overall-9 the default descending sort. | |
| 4. Show Overall-10 only with an explicit "paper metric" label. | |
| 5. Show `N/A` for inapplicable values and never coerce them to zero in | |
| JavaScript. | |
| 6. Display prompt count, scored count, coverage, and missing policy. | |
| 7. Add tooltips with exact metric and multi-label definitions. | |
| 8. Update the title, introduction, metrics, submission text, links, and citation | |
| inherited from BrowseComp-Plus. | |
| ### Phase F: Release workflow | |
| 1. Re-run canonical evaluations when results change. | |
| 2. Run the exporter. | |
| 3. Run aggregate generation and validation tests. | |
| 4. Review the manifest and aggregate diff. | |
| 5. Build the Space and run a local smoke test. | |
| 6. Publish the data artifacts and Space from the same version tag. | |
| ## 11. Required validation checks | |
| The release must fail unless all applicable checks pass: | |
| - Exactly 751 unique prompt IDs | |
| - Partition counts exactly 100/651 and 100/387/264 | |
| - Every tag field is a list with no duplicate value within a prompt | |
| - No duplicate prompt-model record | |
| - Scored component values are within 0–3 | |
| - `null`, missing, invalid, and zero remain distinguishable | |
| - Per-prompt Overall-9 and Overall-10 recompute exactly from their component sets | |
| - Aggregate coverage and denominators reproduce the declared policy | |
| - Overall-10 aggregate values match the canonical recomputation output | |
| - Aggregate values are invariant to input row order | |
| - Frontend renders missing components as `N/A`, not `0` or `NaN` | |
| ## 12. Recommended acceptance criteria | |
| The data layer is ready when a fresh checkout can generate all public JSON | |
| artifacts with one documented command, validation catches an intentionally | |
| removed result, Overall-10 matches the paper computation, and every displayed | |
| Overall-9 value can be traced to a defined set of prompt IDs and non-null | |
| components. | |