LEADERBOARD_DATA_DESIGN.md

SearchGen-Bench Per-Prompt Data and Leaderboard Design

Implementation status

This design is implemented in the Space:

• scripts/build_leaderboard_data.py imports the canonical ToolGen extraction function, exports all prompt-model records, validates them, and generates the four aggregate views.
• scripts/validate_leaderboard_data.py independently reconstructs all committed aggregates from prompt_scores.jsonl without requiring ToolGen.
• public/data/manifest.json pins the schema, input hash, components, model source IDs, partitions, and missing-result policies.
• The frontend provides component/stratum rankings plus domain and failure-mode scoreboards.
• Hugging Face serves the locally built dist/ contents published at the Space repository root and performs no build or mutable runtime data download. A Dockerfile remains available for optional container deployment.

To regenerate a release:

TOOLGEN_ROOT=/path/to/ToolGen npm run build:data
npm run test:data
npm run build

1. Decision

The proposed design is sound: create one versioned record for each of the 751 evaluation prompts, attach its benchmark tags, and record per-model component scores. All public scoreboards should be derived from this data rather than maintained as hand-written tables.

One metric distinction must remain explicit:

• The paper's canonical score is the mean of 10 applicable components.
• The proposed public leaderboard displays nine components by excluding text_reference_evaluation (called "text fidelity" in the proposal).
• text_rendering remains included; it is not the excluded component.
• A mean recomputed from the remaining nine components must be named Overall-9. It is not numerically interchangeable with the paper's canonical Overall-10.

Do not discard the tenth component at extraction time. Preserve all 10 raw components so the exported data can reproduce the paper, then define the nine-component leaderboard as a named metric view. This avoids an irreversible data loss and makes discrepancies auditable.

2. Canonical inputs

Use the sources defined by TABLE_UPDATE_SOP.md:

• Prompt metadata: phase4d1_generations/eval_datasets_548_finalized_v1.jsonl
• Evaluation result: final_20k_release_v2/{release_row}/{lane}/{generator}/augmented_parsed_result_ffjudge_pp.json
• Extraction semantics: reuse extract_10comp() from paper_materials/recompute_tables.py; do not create a second implementation of the component rules.

The markdown SOP is documentation, not a machine-readable input. The JSONL and evaluation result files remain the ground truth.

3. Prompt identity and tags

Use sample_id as the stable primary key. An integer prompt_index may be included for presentation, but it must never be used to join data because file ordering can change.

Each prompt record should contain:

• sample_id
• prompt_index
• release_row
• stratum: NoSearch or SearchIntensive
• search_type: NoSearch, VisualSearch, or TextualSearch
• domains: list of zero or more domain tags
• failure_modes: list of zero or more failure-mode tags
• Optional useful filters: difficulty, language, generation_task_type, and is_miniset

Derive the partitions exactly as specified by the SOP:

if row.get("subset") == "NoSearch":
    stratum = "NoSearch"
    search_type = "NoSearch"
elif "texthard" in row["sample_id"] or "text_rendering" in row["sample_id"]:
    stratum = "SearchIntensive"
    search_type = "TextualSearch"
else:
    stratum = "SearchIntensive"
    search_type = "VisualSearch"

Expected counts are:

Partition	Count
NoSearch	100
SearchIntensive	651
VisualSearch	387
TextualSearch	264
Total	751

4. Score schema

Use a normalized, lossless JSONL artifact for the canonical public data. One prompt record can contain a models map to avoid repeating prompt metadata:

{
  "sample_id": "eval_base40_001",
  "prompt_index": 0,
  "release_row": "evalset_rows/eval_base40_001",
  "stratum": "SearchIntensive",
  "search_type": "VisualSearch",
  "domains": ["CUL", "ART", "TEC", "ARC"],
  "failure_modes": ["TK-R", "EIK", "CS", "THG", "CCS"],
  "models": {
    "gpt_image": {
      "status": "scored",
      "lane": "none",
      "components_raw_0to3": {
        "checklist": 2.1,
        "rubric_adaptive": 2.0,
        "prompt_faithfulness": 2.2,
        "image_quality": 2.4,
        "text_rendering": 2.1,
        "ai_naturalness": 2.0,
        "composition_and_aesthetics": 2.5,
        "physical_plausibility": 2.3,
        "visual_reference_evaluation": 1.9,
        "text_reference_evaluation": null
      },
      "overall_10_raw": 2.1667,
      "overall_9_raw": 2.1667
    }
  }
}

Store unrounded 0–3 values, or values with enough precision to reproduce the aggregates. Convert to 0–100 and round only in derived artifacts or at display time.

null means not applicable. It is different from a missing result and from a real score of zero. Represent result state separately:

• scored: evaluation exists and was parsed
• missing_generation: no generated image/result
• missing_evaluation: image may exist but evaluation is absent
• invalid_evaluation: file exists but is malformed or unusable

5. Metric definitions

The nine displayed components are:

1. Checklist
2. Rubric Adaptive
3. Prompt Faithfulness
4. Image Quality
5. Text Rendering
6. AI Naturalness
7. Composition and Aesthetics
8. Physical Plausibility
9. Visual Reference

The excluded public component is Text Reference: text_reference_evaluation.

For a scored prompt-model pair:

Overall-10 = mean(non-null values among all 10 components)
Overall-9  = mean(non-null values among the nine displayed components)
Public scale = raw score * 100 / 3

For a group such as a domain or failure mode:

Group Overall-9 = mean(per-prompt Overall-9 values included by policy)
Group component = mean(non-null per-prompt values for that component)

This is prompt-macro averaging. Do not pool checklist items or rubric items across prompts, because prompts with more items would receive more weight.

6. Missing-result and coverage policy

Raw data must preserve missing states. The aggregation layer applies the paper's model-specific policy:

• Default: a missing result is zero-filled for aggregate computation.
• xai_image and qwen_image_2_pro: missing generation/evaluation results are excluded from the denominator.
• Inapplicable components remain null and are excluded from the relevant component denominator.

Every aggregate row must publish:

• n_total: prompts in the selected group
• n_scored: prompts with valid evaluations
• n_included: prompts included after applying the missing-result policy
• coverage: n_scored / n_total
• missing_policy: zero_fill or exclude

Rankings should either require a declared minimum coverage or visibly flag low coverage. A high score with materially incomplete coverage must not appear equivalent to a fully covered score.

7. Multi-label breakdown semantics

domains and failure_modes are multi-label fields. A prompt contributes once to every tag assigned to it. Therefore:

• Domain row counts overlap and must not be summed to obtain 751.
• Failure-mode row counts also overlap.
• The UI must label these as multi-label breakdowns.
• Each breakdown row must show its own prompt count and model coverage.

Avoid fractional weighting across tags unless a separate, explicitly named analysis requires it. Full membership is easier to explain and audit.

8. Derived scoreboards

Generate the following views from the same prompt-level artifact.

8.1 Overall model leaderboard

One row per model, ranked by Overall-9. Show:

• Rank and model metadata
• Overall-9
• Overall-10, labeled as the paper-compatible score
• All nine public components
• Coverage and missing policy

8.2 Stratum leaderboard

Provide filters or tabs for:

• All 751
• NoSearch (100)
• SearchIntensive (651)
• VisualSearch (387)
• TextualSearch (264)

Each view shows Overall-9, the nine components, and coverage.

8.3 Domain breakdown

For a selected model, show each domain's Overall-9, prompt count, and coverage. Optionally allow expanding a domain to reveal its nine components. This is more readable than placing every domain-component combination in one wide table.

8.4 Failure-mode breakdown

Use the same layout and aggregation rules as the domain breakdown.

8.5 Comparison view

Allow users to select a small number of models and compare their component profiles within a selected stratum, domain, or failure mode. This should use the same aggregate file rather than recomputing with different rules.

9. Artifact layout

Recommended generated files:

public/data/
├── manifest.json
├── prompt_scores.jsonl
├── leaderboard_overall.json
├── leaderboard_by_stratum.json
├── leaderboard_by_domain.json
└── leaderboard_by_failure_mode.json

manifest.json should include:

• Schema version
• Benchmark dataset version/hash
• Scoring version, including ffjudge_pp and PP configuration
• Generation timestamp
• Source repository commit, when available
• Component lists for Overall-10 and Overall-9
• Scale and rounding rules
• Missing-result policies
• Model display names, types, and ordering metadata

The browser should normally load the compact aggregate JSON files. Ship prompt_scores.jsonl for audit/download or client-side custom exploration, not as a prerequisite for rendering the initial table.

10. Implementation actions

Phase A: Freeze semantics

1. Confirm that "drop text fidelity" means excluding text_reference_evaluation, not text_rendering.
2. Approve Overall-9 as the public primary metric and retain Overall-10 for paper compatibility.
3. Freeze model IDs, display names, model types, lane selection, missing-result policies, and minimum ranking coverage.

Phase B: Build the exporter

1. Add a producer-side script in ToolGen, for example paper_materials/export_searchgen_leaderboard.py.
2. Import or refactor and reuse extract_10comp() rather than copying it.
3. Load all 751 prompt metadata rows and assert unique sample_id values.
4. Derive the two-level partition and copy all list-valued tags.
5. Resolve each configured model's canonical none-lane PP evaluation.
6. Record all 10 component values plus explicit result status.
7. Compute per-prompt Overall-10 and Overall-9.
8. Write the prompt JSONL and manifest deterministically.

The exporter belongs near the canonical ToolGen data. The Hugging Face Space must not attempt to traverse a local ToolGen path at runtime.

Phase C: Build and validate aggregates

1. Add a second script that reads only the exported JSONL and manifest.
2. Apply the declared missing policies and macro-aggregation rules.
3. Produce overall, stratum, domain, and failure-mode JSON files.
4. Sort output keys/rows deterministically to make reviews meaningful.
5. Compare Overall-10 outputs against recompute_tables.py --json before accepting a release.
6. Snapshot expected Overall-9 results in tests.

Phase D: Replace the inherited BrowseComp data path

1. Remove the download of Tevatron/BrowseComp-Plus-results from scripts/extract_data.py.
2. Remove the runtime CSV-to-JSON extraction from the Docker startup command.
3. Generate and commit/copy the versioned SearchGen artifacts into public/data/ before the Vite build, or download one pinned artifact release from a dedicated SearchGen dataset repository.
4. Fail the build if data validation fails; do not deploy an empty leaderboard.

Phase E: Update the frontend

1. Replace the BrowseComp-specific columns in main.js.
2. Add table selectors for overall, strata, domains, and failure modes.
3. Make Overall-9 the default descending sort.
4. Show Overall-10 only with an explicit "paper metric" label.
5. Show N/A for inapplicable values and never coerce them to zero in JavaScript.
6. Display prompt count, scored count, coverage, and missing policy.
7. Add tooltips with exact metric and multi-label definitions.
8. Update the title, introduction, metrics, submission text, links, and citation inherited from BrowseComp-Plus.

Phase F: Release workflow

1. Re-run canonical evaluations when results change.
2. Run the exporter.
3. Run aggregate generation and validation tests.
4. Review the manifest and aggregate diff.
5. Build the Space and run a local smoke test.
6. Publish the data artifacts and Space from the same version tag.

11. Required validation checks

The release must fail unless all applicable checks pass:

• Exactly 751 unique prompt IDs
• Partition counts exactly 100/651 and 100/387/264
• Every tag field is a list with no duplicate value within a prompt
• No duplicate prompt-model record
• Scored component values are within 0–3
• null, missing, invalid, and zero remain distinguishable
• Per-prompt Overall-9 and Overall-10 recompute exactly from their component sets
• Aggregate coverage and denominators reproduce the declared policy
• Overall-10 aggregate values match the canonical recomputation output
• Aggregate values are invariant to input row order
• Frontend renders missing components as N/A, not 0 or NaN

12. Recommended acceptance criteria

The data layer is ready when a fresh checkout can generate all public JSON artifacts with one documented command, validation catches an intentionally removed result, Overall-10 matches the paper computation, and every displayed Overall-9 value can be traced to a defined set of prompt IDs and non-null components.