Genre-representation benchmark checklist

This checklist preregisters the test of whether a redesigned learned representation captures more than elementary formula-composition statistics. Complete it for every reported run; do not tune the rule after viewing held-out results.

Freeze before evaluation

Freeze the dataset version, genre labels, model checkpoint, random seeds, and split manifest.
Use at least three independently seeded held-out splits.
Keep all active molecule identities disjoint between train and held-out records.
Keep exact formulas disjoint between train and held-out records.
Fit normalization, centroids, and any other learned preprocessing on training records only.
Store learned train/test representations in manifest-aligned .npy files without genre labels as inputs.

Run the controlled comparison

Run scripts/genre_baseline_benchmark.py on the frozen manifest.
Confirm every leakage_audit.passed value is true; a failure invalidates the split rather than becoming an exclusion after results are seen.
Compare both representations with the identical nearest-centroid probe and identical held-out rows.
Report held-out accuracy for the learned representation and composition baseline.
Report the paired bootstrap accuracy delta and 95% confidence interval for every split.
Archive the manifest, output JSON, checkpoint identifier, and commit hash.

Decision and reporting

Claim support only if the lower 95% bound for learned-minus-baseline accuracy is above the preregistered margin on every valid split and at least three splits are valid.
If that criterion fails, report the result as negative; do not substitute a favorable in-sample or leaky split.
In a negative report, state the supported interpretation: the benchmark's genre structure may be explained by elementary compositional statistics, and the experiment does not establish deeper formula semantics.
Report all split results, including weak or adverse ones, and document any invalid split with its leakage audit.

Manifest format:

[
  {
    "name": "seed_1",
    "train_records": "data/splits/seed_1_train.jsonl",
    "test_records": "data/splits/seed_1_test.jsonl",
    "learned_train": "artifacts/seed_1_train_embeddings.npy",
    "learned_test": "artifacts/seed_1_test_embeddings.npy"
  }
]

Example:

python scripts/genre_baseline_benchmark.py \
  --manifest artifacts/genre_split_manifest.json \
  --output artifacts/genre_baseline_report.json \
  --minimum-splits 3 --margin 0