GENRE_BENCHMARK_CHECKLIST.md

# Genre-representation benchmark checklist

This checklist preregisters the test of whether a redesigned learned representation captures more than elementary formula-composition statistics. Complete it for every reported run; do not tune the rule after viewing held-out results.

## Freeze before evaluation

- [ ] Freeze the dataset version, genre labels, model checkpoint, random seeds, and split manifest.
- [ ] Use at least three independently seeded held-out splits.
- [ ] Keep all active molecule identities disjoint between train and held-out records.
- [ ] Keep exact formulas disjoint between train and held-out records.
- [ ] Fit normalization, centroids, and any other learned preprocessing on training records only.
- [ ] Store learned train/test representations in manifest-aligned `.npy` files without genre labels as inputs.

## Run the controlled comparison

- [ ] Run `scripts/genre_baseline_benchmark.py` on the frozen manifest.
- [ ] Confirm every `leakage_audit.passed` value is `true`; a failure invalidates the split rather than becoming an exclusion after results are seen.
- [ ] Compare both representations with the identical nearest-centroid probe and identical held-out rows.
- [ ] Report held-out accuracy for the learned representation and composition baseline.
- [ ] Report the paired bootstrap accuracy delta and 95% confidence interval for every split.
- [ ] Archive the manifest, output JSON, checkpoint identifier, and commit hash.

## Decision and reporting

- [ ] Claim support only if the lower 95% bound for learned-minus-baseline accuracy is above the preregistered margin on every valid split and at least three splits are valid.
- [ ] If that criterion fails, report the result as negative; do not substitute a favorable in-sample or leaky split.
- [ ] In a negative report, state the supported interpretation: the benchmark's genre structure may be explained by elementary compositional statistics, and the experiment does not establish deeper formula semantics.
- [ ] Report all split results, including weak or adverse ones, and document any invalid split with its leakage audit.

Manifest format:

```json
[
  {
    "name": "seed_1",
    "train_records": "data/splits/seed_1_train.jsonl",
    "test_records": "data/splits/seed_1_test.jsonl",
    "learned_train": "artifacts/seed_1_train_embeddings.npy",
    "learned_test": "artifacts/seed_1_test_embeddings.npy"
  }
]
```

Example:

```bash
python scripts/genre_baseline_benchmark.py \
  --manifest artifacts/genre_split_manifest.json \
  --output artifacts/genre_baseline_report.json \
  --minimum-splits 3 --margin 0
```