File size: 2,573 Bytes
bd082fe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | # Main_benchmark benchmark — scoring scripts + reference structures
Two-state / multi-conformer benchmark of 60 cases across four patterns
(FS fold-switch, AL allosteric, ID disorder-to-order, OL oligomer), 6 MSA-subsampling
methods each. This package ships the **scoring code** and the **per-case reference
structures + region definitions** so predictions can be scored and the benchmark
tables regenerated on CPU. The AlphaFold2 prediction PDBs (~16 GB) are **not**
distributed here — bring your own, or score the reference set.
## Contents
- `eval/main_benchmark_evaluate.py` — per-prediction scorer. For each state (a, b):
Cα RMSD on the common core, optional TM-score, and `hit_primary` (§9.1 binding
3-condition: RMSD ≤ 3 Å on common core AND mean pLDDT ≥ 70 AND switch-region
pLDDT ≥ 70). Pure CPU: `numpy`, `biopython`, `pyyaml`. TMalign optional (extra
columns only).
- `eval/aggregate_main_benchmark.py` — aggregate per-(case, method) `evals.tsv` into
`main_table.csv` (per case×method hit counts/rates) and `pattern_summary.csv`
(per pattern×method means).
- `bench/main_benchmark/structures/<case_id>/{state_a,state_b}.pdb` — 60 cases, 120 references.
- `bench/main_benchmark/annotations/<case_id>/state_region_FINAL.tsv` — common-core and
switch-region residue indices (evaluator input only; oracle annotation, not a feature).
- `bench/main_benchmark/cases.yaml` — case metadata.
## Usage
```bash
pip install numpy biopython pyyaml # TMalign optional
# Score one case×method directory of prediction PDBs:
python eval/main_benchmark_evaluate.py \
--case SFB_FS_A1AT_1QLPA_1OPHA \
--root /path/to/predictions/<case>/<method>/refine_per_state \
--out evals.tsv
# Aggregate a full results tree (SF_MAIN_BENCH_RESULTS/<case>/<method>/refine_per_state/evals.tsv):
SF_MAIN_BENCH_RESULTS=/path/to/results python eval/aggregate_main_benchmark.py
```
`SF_MAIN_BENCH_DATA` defaults to the bundled `bench/main_benchmark` (structures + annotations +
cases.yaml); override to point elsewhere. `SF_MAIN_BENCH_RESULTS` supplies the predictions tree.
## Verification
The scorer is deterministic. Re-scoring the internal prediction set reproduces the
published hit flags exactly (2594/2594 predictions across a 12-case stratified sample;
`main_table` → `pattern_summary` aggregation reproduces 24/24 rows; `main_table` hit
counts reproduce from `evals.tsv` 360/360). Headline: mosaic-SF beats AF-Cluster by
~5–18 pp across patterns but is statistically indistinguishable from depth-matched
random subsampling.
|