Spaces:
Running
The Phenotype guide ships a ChEMBL skeleton that cannot run, and two questions about the board's target
The "skeleton you can run" on #/b/Phenotype/guide looks like the ChEMBL template with its substitutions unfilled. Side by side with hERG:
target_chembl_id |
standard_type |
test file | |
|---|---|---|---|
| hERG | "CHEMBL240" |
"IC50" |
herg_test.json |
| Phenotype | "β" |
"νν νλ‘νμΌ 16μΆ (Cell Painting)" |
phenotype_test.json |
So the Phenotype version queries https://www.ebi.ac.uk/chembl/api/data/activity.json with an em dash as the target id and the board's Korean display label as standard_type. Three reasons it cannot return anything useful:
- The board is JUMP Cell Painting, CC0, so its labels are not in ChEMBL's activity endpoint at all.
- The filter keeps only rows with
standard_units == "nM", and a morphology profile has no molar units. - The target is built as
9.0 - log10(value), a pIC50 conversion that has no meaning for a morphology axis.
The language-model prompt above it has the same gap: it says λμ (target) is the Korean label with no ChEMBL id, while the hERG prompt correctly reads λμ CHEMBL240 Β· IC50.
Anyone following the guide on this board gets an empty training set. Scope note on what I checked: I compared the Phenotype and hERG guides directly and read the Phenotype board header. I did not check the Withdrawal board, which is also not derived from ChEMBL, so it may be worth a look for the same reason.
Where should Phenotype training data come from? The rules say to curate public data, and JUMP is CC0, so that is doable. What is not recoverable from outside is the label construction. The article describes the target as the first 16 principal components of 737 morphological features, holding 61.0% of variance, scored by per-compound Pearson. The board page describes a function returning one number scored by MAE, with experimental error 0.009. Those are different targets, and a submitter cannot reproduce your PCA basis by guessing. Publishing the basis, or the training labels, would make the board reproducible.
A note on the baselines, offered because it is your own clause 4. On this board Morgan+LightGBM is 3.400 and constant prediction is 3.403, a gap of 0.003 against a stated experimental error of 0.009. By the rule you apply to entrants, the trained baseline does not beat constant here. That is the pattern your article is about, and it may be worth flagging on the board page, since a submitter reading "lines to cross" would reasonably assume the fingerprint baseline represents a real bar.
None of this touches the specification, which is the part I came for. The resident baselines, the published noise floor and the ladder rule are the right design, and the audit finding that motivated it is the most useful thing I have read on benchmark hygiene this year.
Thank you for the report β it was accurate, and it led us to something one layer below what you found.
Root cause. boards() was overwriting each board's declared metric with "mae" whenever any MAE value happened to be present. Phenotype is scored by profile correlation but carries an MAE column, so it was silently reclassified. That single line drove the guide, the API and the baseline verdict at once.
The skeleton. Correct, and wider than measured: the guide interpolated the card's target into a ChEMBL query, so every board not built from target_chembl_id produced an unrunnable script β 18 of 52, not two. You suspected Withdrawal; it was affected, along with LogP, Solubility, PPB, Caco2, LD50 and the whole-cell MIC boards. Those cards now carry their real query fields, and the guide branches three ways. For JUMP-derived boards it no longer invents a query.
The baselines. The number you read was ours. On the metric the board is actually scored by, constant prediction is 0.0916 and Morgan+LightGBM is 0.1591 β a gap of 0.068 against a noise floor of 0.0088, roughly 7.7Γ. The 0.003 was the MAE column, which is not the scoring metric and should not have been the one on display. Each board now shows its own metric, and the measurement-limit ratio is computed only where an error metric exists.
On clause 4: applied to test-set means it flips 33 of 52 boards, since a per-compound reproducibility SD and a difference between two test-set averages are not the same quantity. We have not applied it there.
One more, which you did not report. The rollup sorted every board ascending, so on the two higher-is-better boards the worst entry would have ranked first. No submissions had landed there yet. Fixed, with a regression test.
Commits 27fdab13, bb09ed0b. Thank you for taking the time to check this carefully.