# SearchGen-Bench Per-Prompt Data and Leaderboard Design

## Implementation status

This design is implemented in the Space:

- `scripts/build_leaderboard_data.py` imports the canonical ToolGen extraction
  function, exports all prompt-model records, validates them, and generates the
  four aggregate views.
- `scripts/validate_leaderboard_data.py` independently reconstructs all
  committed aggregates from `prompt_scores.jsonl` without requiring ToolGen.
- `public/data/manifest.json` pins the schema, input hash, components, model
  source IDs, partitions, and missing-result policies.
- The frontend provides component/stratum rankings plus domain and failure-mode
  scoreboards.
- Hugging Face serves the locally built `dist/` contents published at the Space
  repository root and performs no build or mutable runtime data download. A
  Dockerfile remains available for optional container deployment.

To regenerate a release:

```bash
TOOLGEN_ROOT=/path/to/ToolGen npm run build:data
npm run test:data
npm run build
```

## 1. Decision

The proposed design is sound: create one versioned record for each of the 751
evaluation prompts, attach its benchmark tags, and record per-model component
scores. All public scoreboards should be derived from this data rather than
maintained as hand-written tables.

One metric distinction must remain explicit:

- The paper's canonical score is the mean of 10 applicable components.
- The proposed public leaderboard displays nine components by excluding
  `text_reference_evaluation` (called "text fidelity" in the proposal).
- `text_rendering` remains included; it is not the excluded component.
- A mean recomputed from the remaining nine components must be named
  **Overall-9**. It is not numerically interchangeable with the paper's
  canonical **Overall-10**.

Do not discard the tenth component at extraction time. Preserve all 10 raw
components so the exported data can reproduce the paper, then define the
nine-component leaderboard as a named metric view. This avoids an irreversible
data loss and makes discrepancies auditable.

## 2. Canonical inputs

Use the sources defined by `TABLE_UPDATE_SOP.md`:

- Prompt metadata:
  `phase4d1_generations/eval_datasets_548_finalized_v1.jsonl`
- Evaluation result:
  `final_20k_release_v2/{release_row}/{lane}/{generator}/augmented_parsed_result_ffjudge_pp.json`
- Extraction semantics: reuse `extract_10comp()` from
  `paper_materials/recompute_tables.py`; do not create a second implementation
  of the component rules.

The markdown SOP is documentation, not a machine-readable input. The JSONL and
evaluation result files remain the ground truth.

## 3. Prompt identity and tags

Use `sample_id` as the stable primary key. An integer `prompt_index` may be
included for presentation, but it must never be used to join data because file
ordering can change.

Each prompt record should contain:

- `sample_id`
- `prompt_index`
- `release_row`
- `stratum`: `NoSearch` or `SearchIntensive`
- `search_type`: `NoSearch`, `VisualSearch`, or `TextualSearch`
- `domains`: list of zero or more domain tags
- `failure_modes`: list of zero or more failure-mode tags
- Optional useful filters: `difficulty`, `language`,
  `generation_task_type`, and `is_miniset`

Derive the partitions exactly as specified by the SOP:

```python
if row.get("subset") == "NoSearch":
    stratum = "NoSearch"
    search_type = "NoSearch"
elif "texthard" in row["sample_id"] or "text_rendering" in row["sample_id"]:
    stratum = "SearchIntensive"
    search_type = "TextualSearch"
else:
    stratum = "SearchIntensive"
    search_type = "VisualSearch"
```

Expected counts are:

| Partition | Count |
|---|---:|
| NoSearch | 100 |
| SearchIntensive | 651 |
| VisualSearch | 387 |
| TextualSearch | 264 |
| Total | 751 |

## 4. Score schema

Use a normalized, lossless JSONL artifact for the canonical public data. One
prompt record can contain a `models` map to avoid repeating prompt metadata:

```json
{
  "sample_id": "eval_base40_001",
  "prompt_index": 0,
  "release_row": "evalset_rows/eval_base40_001",
  "stratum": "SearchIntensive",
  "search_type": "VisualSearch",
  "domains": ["CUL", "ART", "TEC", "ARC"],
  "failure_modes": ["TK-R", "EIK", "CS", "THG", "CCS"],
  "models": {
    "gpt_image": {
      "status": "scored",
      "lane": "none",
      "components_raw_0to3": {
        "checklist": 2.1,
        "rubric_adaptive": 2.0,
        "prompt_faithfulness": 2.2,
        "image_quality": 2.4,
        "text_rendering": 2.1,
        "ai_naturalness": 2.0,
        "composition_and_aesthetics": 2.5,
        "physical_plausibility": 2.3,
        "visual_reference_evaluation": 1.9,
        "text_reference_evaluation": null
      },
      "overall_10_raw": 2.1667,
      "overall_9_raw": 2.1667
    }
  }
}
```

Store unrounded 0–3 values, or values with enough precision to reproduce the
aggregates. Convert to 0–100 and round only in derived artifacts or at display
time.

`null` means not applicable. It is different from a missing result and from a
real score of zero. Represent result state separately:

- `scored`: evaluation exists and was parsed
- `missing_generation`: no generated image/result
- `missing_evaluation`: image may exist but evaluation is absent
- `invalid_evaluation`: file exists but is malformed or unusable

## 5. Metric definitions

The nine displayed components are:

1. Checklist
2. Rubric Adaptive
3. Prompt Faithfulness
4. Image Quality
5. Text Rendering
6. AI Naturalness
7. Composition and Aesthetics
8. Physical Plausibility
9. Visual Reference

The excluded public component is Text Reference:
`text_reference_evaluation`.

For a scored prompt-model pair:

```text
Overall-10 = mean(non-null values among all 10 components)
Overall-9  = mean(non-null values among the nine displayed components)
Public scale = raw score * 100 / 3
```

For a group such as a domain or failure mode:

```text
Group Overall-9 = mean(per-prompt Overall-9 values included by policy)
Group component = mean(non-null per-prompt values for that component)
```

This is prompt-macro averaging. Do not pool checklist items or rubric items
across prompts, because prompts with more items would receive more weight.

## 6. Missing-result and coverage policy

Raw data must preserve missing states. The aggregation layer applies the paper's
model-specific policy:

- Default: a missing result is zero-filled for aggregate computation.
- `xai_image` and `qwen_image_2_pro`: missing generation/evaluation results are
  excluded from the denominator.
- Inapplicable components remain `null` and are excluded from the relevant
  component denominator.

Every aggregate row must publish:

- `n_total`: prompts in the selected group
- `n_scored`: prompts with valid evaluations
- `n_included`: prompts included after applying the missing-result policy
- `coverage`: `n_scored / n_total`
- `missing_policy`: `zero_fill` or `exclude`

Rankings should either require a declared minimum coverage or visibly flag low
coverage. A high score with materially incomplete coverage must not appear
equivalent to a fully covered score.

## 7. Multi-label breakdown semantics

`domains` and `failure_modes` are multi-label fields. A prompt contributes once
to every tag assigned to it. Therefore:

- Domain row counts overlap and must not be summed to obtain 751.
- Failure-mode row counts also overlap.
- The UI must label these as multi-label breakdowns.
- Each breakdown row must show its own prompt count and model coverage.

Avoid fractional weighting across tags unless a separate, explicitly named
analysis requires it. Full membership is easier to explain and audit.

## 8. Derived scoreboards

Generate the following views from the same prompt-level artifact.

### 8.1 Overall model leaderboard

One row per model, ranked by Overall-9. Show:

- Rank and model metadata
- Overall-9
- Overall-10, labeled as the paper-compatible score
- All nine public components
- Coverage and missing policy

### 8.2 Stratum leaderboard

Provide filters or tabs for:

- All 751
- NoSearch (100)
- SearchIntensive (651)
- VisualSearch (387)
- TextualSearch (264)

Each view shows Overall-9, the nine components, and coverage.

### 8.3 Domain breakdown

For a selected model, show each domain's Overall-9, prompt count, and coverage.
Optionally allow expanding a domain to reveal its nine components. This is more
readable than placing every domain-component combination in one wide table.

### 8.4 Failure-mode breakdown

Use the same layout and aggregation rules as the domain breakdown.

### 8.5 Comparison view

Allow users to select a small number of models and compare their component
profiles within a selected stratum, domain, or failure mode. This should use the
same aggregate file rather than recomputing with different rules.

## 9. Artifact layout

Recommended generated files:

```text
public/data/
├── manifest.json
├── prompt_scores.jsonl
├── leaderboard_overall.json
├── leaderboard_by_stratum.json
├── leaderboard_by_domain.json
└── leaderboard_by_failure_mode.json
```

`manifest.json` should include:

- Schema version
- Benchmark dataset version/hash
- Scoring version, including `ffjudge_pp` and PP configuration
- Generation timestamp
- Source repository commit, when available
- Component lists for Overall-10 and Overall-9
- Scale and rounding rules
- Missing-result policies
- Model display names, types, and ordering metadata

The browser should normally load the compact aggregate JSON files. Ship
`prompt_scores.jsonl` for audit/download or client-side custom exploration, not
as a prerequisite for rendering the initial table.

## 10. Implementation actions

### Phase A: Freeze semantics

1. Confirm that "drop text fidelity" means excluding
   `text_reference_evaluation`, not `text_rendering`.
2. Approve `Overall-9` as the public primary metric and retain `Overall-10` for
   paper compatibility.
3. Freeze model IDs, display names, model types, lane selection, missing-result
   policies, and minimum ranking coverage.

### Phase B: Build the exporter

1. Add a producer-side script in ToolGen, for example
   `paper_materials/export_searchgen_leaderboard.py`.
2. Import or refactor and reuse `extract_10comp()` rather than copying it.
3. Load all 751 prompt metadata rows and assert unique `sample_id` values.
4. Derive the two-level partition and copy all list-valued tags.
5. Resolve each configured model's canonical `none`-lane PP evaluation.
6. Record all 10 component values plus explicit result status.
7. Compute per-prompt Overall-10 and Overall-9.
8. Write the prompt JSONL and manifest deterministically.

The exporter belongs near the canonical ToolGen data. The Hugging Face Space
must not attempt to traverse a local ToolGen path at runtime.

### Phase C: Build and validate aggregates

1. Add a second script that reads only the exported JSONL and manifest.
2. Apply the declared missing policies and macro-aggregation rules.
3. Produce overall, stratum, domain, and failure-mode JSON files.
4. Sort output keys/rows deterministically to make reviews meaningful.
5. Compare Overall-10 outputs against `recompute_tables.py --json` before
   accepting a release.
6. Snapshot expected Overall-9 results in tests.

### Phase D: Replace the inherited BrowseComp data path

1. Remove the download of `Tevatron/BrowseComp-Plus-results` from
   `scripts/extract_data.py`.
2. Remove the runtime CSV-to-JSON extraction from the Docker startup command.
3. Generate and commit/copy the versioned SearchGen artifacts into
   `public/data/` before the Vite build, or download one pinned artifact release
   from a dedicated SearchGen dataset repository.
4. Fail the build if data validation fails; do not deploy an empty leaderboard.

### Phase E: Update the frontend

1. Replace the BrowseComp-specific columns in `main.js`.
2. Add table selectors for overall, strata, domains, and failure modes.
3. Make Overall-9 the default descending sort.
4. Show Overall-10 only with an explicit "paper metric" label.
5. Show `N/A` for inapplicable values and never coerce them to zero in
   JavaScript.
6. Display prompt count, scored count, coverage, and missing policy.
7. Add tooltips with exact metric and multi-label definitions.
8. Update the title, introduction, metrics, submission text, links, and citation
   inherited from BrowseComp-Plus.

### Phase F: Release workflow

1. Re-run canonical evaluations when results change.
2. Run the exporter.
3. Run aggregate generation and validation tests.
4. Review the manifest and aggregate diff.
5. Build the Space and run a local smoke test.
6. Publish the data artifacts and Space from the same version tag.

## 11. Required validation checks

The release must fail unless all applicable checks pass:

- Exactly 751 unique prompt IDs
- Partition counts exactly 100/651 and 100/387/264
- Every tag field is a list with no duplicate value within a prompt
- No duplicate prompt-model record
- Scored component values are within 0–3
- `null`, missing, invalid, and zero remain distinguishable
- Per-prompt Overall-9 and Overall-10 recompute exactly from their component sets
- Aggregate coverage and denominators reproduce the declared policy
- Overall-10 aggregate values match the canonical recomputation output
- Aggregate values are invariant to input row order
- Frontend renders missing components as `N/A`, not `0` or `NaN`

## 12. Recommended acceptance criteria

The data layer is ready when a fresh checkout can generate all public JSON
artifacts with one documented command, validation catches an intentionally
removed result, Overall-10 matches the paper computation, and every displayed
Overall-9 value can be traced to a defined set of prompt IDs and non-null
components.