artifacts/genre_challenge_v2_release/PREREGISTRATION.md

# Genre Challenge v2: blinded evaluation preregistration

Status: frozen before model predictions or challenge-set embeddings are generated or inspected.

## Experimental unit and systems

The primary aggregation unit is the matched block: 50 blocks, each containing one formula from each of four genres. The frozen systems are:

1. **Composition baseline:** a classifier using only active-component count, elemental mass vector, concentration HHI, top-weight share, and natural-material weight share.
2. **Frozen PIMT representation:** the fixed pretrained PIMT checkpoint, with a linear multinomial probe trained only inside each training fold.
3. **Combined model:** the same probe using composition features plus the frozen PIMT representation.

No challenge labels may be used to select pooling, layers, features, hyperparameters, thresholds, prompts, or checkpoints. All transformations and hyperparameters must be fitted in the corresponding training data only. The exact checkpoint and evaluation-code hashes must be added to the run manifest before predictions are generated.

## Primary endpoint and estimand

The primary endpoint is the block-averaged improvement in four-class balanced accuracy of the combined model over the composition baseline. Accuracy is calculated within each complete matched block and then averaged equally over the 50 blocks. The primary contrast is combined minus composition baseline.

The primary uncertainty interval is a two-sided 95% percentile interval from 10,000 paired bootstrap resamples of the 50 blocks, using seed `8675309`. The confirmatory result is positive only if the interval's lower bound is greater than zero. Otherwise it is negative; there is no post-hoc "trend" category.

## Secondary endpoints

- Block-averaged macro F1 for each system.
- PIMT-only minus composition-baseline balanced accuracy.
- Combined minus composition-baseline log loss.
- Per-class recall and the full confusion matrix.
- Same-genre retrieval mean reciprocal rank and enrichment within matched candidates.
- Real-versus-decoy paired accuracy and ROC AUC when the preregistered decoy tasks are materialized.

Secondary intervals use the same paired 10,000-resample block bootstrap and are descriptive; no multiplicity-adjusted confirmatory claims will be made from them.

## Splitting, exclusions, and leakage controls

Outer folds must keep exact formulas, active ingredient identities, and all variants of a source formula together. Training and test ingredient identities must be disjoint. Preprocessing, probe fitting, calibration, and hyperparameter selection occur inside the training partition. The blinded evaluator payload contains no genre, source index, or formula ID.

All 50 blocks are included. A record may be excluded only for a deterministic execution failure documented before unblinding (for example, an invalid tensor or unavailable frozen checkpoint); its entire four-record block is then excluded from every system and metric. No exclusion based on confidence, error direction, or model output is permitted. If fewer than 45 complete blocks remain, the run is invalid and no confirmatory verdict is issued.

## Ties, missingness, and stopping

Class-score ties are resolved by the fixed lexical class order `amber_oriental`, `citrus_cologne`, `floral_woody`, `fougere`. Retrieval-score ties receive average rank. Missing predictions invalidate the whole block. There is one confirmatory run and no optional stopping, repeated-seed selection, or dataset revision after unblinding.

Predictions, run configuration, dependency snapshot, checkpoint hashes, and evaluation-code commit must be hashed and made read-only before the restricted label key is opened. Any analysis not specified here is labeled exploratory and cannot change the primary verdict.