SeaWolf-AI commited on
Commit
fc23fc4
Β·
verified Β·
1 Parent(s): 4a8b66b

docs: English README with searchable title

Browse files
Files changed (1) hide show
  1. README.md +142 -27
README.md CHANGED
@@ -1,27 +1,142 @@
1
- ---
2
- title: LEADBOARD
3
- emoji: 🎯
4
- colorFrom: blue
5
- colorTo: gray
6
- sdk: docker
7
- app_port: 7860
8
- # hf_oauth λ₯Ό μΌœμ•Ό ν”Œλž«νΌμ΄ OAUTH_CLIENT_ID/SECRET λ₯Ό μ»¨ν…Œμ΄λ„ˆμ— λ„£μ–΄μ€€λ‹€.
9
- hf_oauth: true
10
- pinned: false
11
- short_description: One yardstick for drug prediction tools
12
- tags:
13
- - drug-discovery
14
- - admet
15
- - leaderboard
16
- - benchmark
17
- - cheminformatics
18
- ---
19
-
20
- # LEADBOARD
21
-
22
- μ‹ μ•½ 예츑 도ꡬλ₯Ό, λΆ„μ•Όλ³„λ‘œ, 같은 μž£λŒ€λ‘œ μž°λ‹€.
23
-
24
- 정닡은 λ°°ν¬ν•˜μ§€ μ•ŠλŠ”λ‹€. λŒ€μ‹  μ±„μ μžκ°€ λ¨Όμ € κ³΅κ°œν•œλ‹€ β€”
25
- 아무것도 ν•™μŠ΅ν•˜μ§€ μ•Šμ€ κΈ°μ€€μ„ , κ·Έ λΆ„μ•Όμ˜ μ‹€ν—˜ 작음 λ°”λ‹₯, μ μˆ˜λ§ˆλ‹€μ˜ 데이터·채점기 μ§€λ¬Έ.
26
-
27
- 규격 v1.1 (8개 μ‘°ν•­) Β· FINAL-Bench / VIDRAFT
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: LEADBOARD - ADMET, Kinase and Toxicity Prediction Benchmark
3
+ emoji: 🎯
4
+ colorFrom: blue
5
+ colorTo: gray
6
+ sdk: docker
7
+ app_port: 7860
8
+ # hf_oauth λ₯Ό μΌœμ•Ό ν”Œλž«νΌμ΄ OAUTH_CLIENT_ID/SECRET λ₯Ό μ»¨ν…Œμ΄λ„ˆμ— λ„£μ–΄μ€€λ‹€.
9
+ hf_oauth: true
10
+ pinned: false
11
+ short_description: Benchmark for drug prediction tools - ADMET, kinase, tox
12
+ tags:
13
+ - drug-discovery
14
+ - admet
15
+ - leaderboard
16
+ - benchmark
17
+ - cheminformatics
18
+ - molecular-property-prediction
19
+ - qsar
20
+ - toxicity
21
+ - herg
22
+ - solubility
23
+ - kinase
24
+ - chembl
25
+ - cell-painting
26
+ - preclinical
27
+ ---
28
+
29
+ # LEADBOARD
30
+
31
+ **A benchmark for drug property prediction tools.** One yardstick, one discipline at a time.
32
+
33
+ Submit predictions for a held-out set of molecules. We score them against labels
34
+ we never release, and place your tool on the board for that discipline.
35
+
36
+ πŸ‘‰ **[Open the leaderboard](https://huggingface.co/spaces/FINAL-Bench/leadboard)**
37
+
38
+ ---
39
+
40
+ ## What is measured
41
+
42
+ Most published benchmarks split their molecules at random. That is the wrong
43
+ question. A tool is used to rank compounds nobody has measured yet, so the honest
44
+ test is: *train on what was known by a cut-off year, predict what came after.*
45
+
46
+ We ran both splits on identical data with an identical model. On the hERG
47
+ cardiotoxicity board, AUROC was **0.818** under a random split and **0.606** under
48
+ a time split. Same molecules, same code β€” only the question changed. Every board
49
+ here uses the harder one.
50
+
51
+ ## What we publish before you submit
52
+
53
+ For each board, before any entry arrives:
54
+
55
+ | Published | Why it matters |
56
+ |---|---|
57
+ | **Untrained baselines** | Constant prediction, nearest neighbour, Morgan+LightGBM. If a tool cannot beat these, the board says so. |
58
+ | **Experimental noise floor** | How far apart two labs land when they measure the same compound. Nothing can be measured below it. |
59
+ | **Split grade and answer grade** | Exactly how the test set was cut, and whether the labels can be looked up anywhere. |
60
+ | **Data and scorer fingerprints** | The SHA of the exact test file and scoring code behind every score. |
61
+
62
+ A score without its noise floor is a number without a unit. Both are shown.
63
+
64
+ ## Grading notation
65
+
66
+ Each board carries two letters, for example `[T/P2]`.
67
+
68
+ **Split grade** β€” what the board asks of a tool
69
+ - `T` time split by first-report year
70
+ - `S` scaffold split by Murcko core
71
+ - `R` random split (we do not open these)
72
+
73
+ **Answer grade** β€” whether an entrant can look the answer up
74
+ - `P1` public source, our curation
75
+ - `P2` public source, our unit conversion and selection define this revision
76
+ - `P3` labels held privately
77
+ - `P4` prospective β€” the answer does not exist yet
78
+
79
+ ## Boards
80
+
81
+ 21 boards across 7 disciplines, 18,382 held-out compounds.
82
+
83
+ - πŸ«— **Absorption** β€” Solubility
84
+ - πŸ”₯ **Metabolism** β€” CYP3A4, CYP2D6, CYP2C9
85
+ - ☠️ **Toxicity** β€” hERG
86
+ - 🎯 **Potency** β€” AChE, MAOB, COX2
87
+ - πŸ”‘ **Kinase** β€” EGFR, JAK2, PI3KΞ±, FLT3, VEGFR2, CDK2, HER2, ABL1, BRAF, KIT, ALK
88
+ - πŸ”¬ **Cell / Phenotype** β€” JUMP Cell Painting morphology, 115,689 compounds, 16 profile axes
89
+ - πŸ₯ **Clinical** β€” Post-marketing withdrawal, year-matched controls
90
+
91
+ The withdrawal board is worth a note. Withdrawal rate tracks approval decade
92
+ (7.8% in the 1990s, 1.2% in the 2010s), so a predictor that reads nothing but the
93
+ approval year scores AUROC 0.636. After year-matching the controls it scores 0.504
94
+ β€” chance. That is the version we opened.
95
+
96
+ ## How to enter
97
+
98
+ 1. Pick a board and download its test set β€” structures only, no labels.
99
+ 2. Predict with your own tool. Anything goes: a trained model, a physics engine,
100
+ a language model, a rule of thumb.
101
+ 3. Upload a CSV of `compound_id,prediction`. Sign in with your Hugging Face account.
102
+ 4. Scoring runs off-platform on hardware that holds the labels. The Space never
103
+ sees them.
104
+
105
+ The Space carries a filled-in prompt and a runnable skeleton for each board, so a
106
+ first entry does not require building anything from scratch.
107
+
108
+ ### Repeated submissions
109
+
110
+ New scores are revealed under the ladder rule (Blum & Hardt, ICML 2015): a fresh
111
+ score replaces your best only when it beats it by more than the noise floor.
112
+ Otherwise your previous best stands. This is what stops a leaderboard from being
113
+ won by whoever submits the most times.
114
+
115
+ ## Data and licences
116
+
117
+ - **ChEMBL 37** (EMBL-EBI) β€” CC BY-SA 3.0. Re-curated; attribution travels with every board card.
118
+ - **JUMP Cell Painting Consortium** β€” CC0.
119
+ - Withdrawal board assembled from public regulatory records.
120
+
121
+ Non-commercial research benchmark. Test sets carry structures only; labels are
122
+ never distributed.
123
+
124
+ ---
125
+
126
+ ## ν•œκ΅­μ–΄ μš”μ•½
127
+
128
+ **μ‹ μ•½ 예츑 도ꡬλ₯Ό, λΆ„μ•Όλ³„λ‘œ, 같은 μž£λŒ€λ‘œ μž½λ‹ˆλ‹€.**
129
+
130
+ 정닡은 λ°°ν¬ν•˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€. λŒ€μ‹  μ±„μ μžκ°€ λ¨Όμ € κ³΅κ°œν•©λ‹ˆλ‹€ β€” 아무것도 ν•™μŠ΅ν•˜μ§€ μ•Šμ€
131
+ κΈ°μ€€μ„ , κ·Έ λΆ„μ•Όμ˜ μ‹€ν—˜ 작음 λ°”λ‹₯, μ μˆ˜λ§ˆλ‹€μ˜ 데이터·채점기 μ§€λ¬Έ.
132
+
133
+ λ¬΄μž‘μœ„ 뢄할은 ν‹€λ¦° μ§ˆλ¬Έμž…λ‹ˆλ‹€. λ„κ΅¬λŠ” 아직 아무도 μž¬μ§€ μ•Šμ€ ν™”ν•©λ¬Όμ˜ μˆœμœ„λ₯Ό λ§€κΈ°λŠ”
134
+ 데 μ“°μ΄λ‹ˆ, μ •μ§ν•œ μ‹œν—˜μ€ **μ–΄λŠ ν•΄κΉŒμ§€ μ•Œλ €μ§„ κ²ƒμœΌλ‘œ 배우고 κ·Έ 뒀에 λ‚˜μ˜¨ 것을
135
+ λ§žνžˆλŠ” 것**μž…λ‹ˆλ‹€. 같은 μžλ£ŒΒ·κ°™μ€ λͺ¨λΈλ‘œ 재 보면 hERG 심μž₯독성 AUROC κ°€ λ¬΄μž‘μœ„
136
+ λΆ„ν•  0.818, μ‹œκ°„ λΆ„ν•  0.606 μ΄μ—ˆμŠ΅λ‹ˆλ‹€. μ—¬κΈ° 뢀문은 μ „λΆ€ μ–΄λ €μš΄ μͺ½μ„ μ”λ‹ˆλ‹€.
137
+
138
+ 7개 λΆ„μ•Ό 21λΆ€λ¬Έ Β· ν…ŒμŠ€νŠΈ ν™”ν•©λ¬Ό 18,382. 화면은 ν•œκ΅­μ–΄Β·μ˜μ–΄λ₯Ό μžλ™μœΌλ‘œ κ°€λ¦…λ‹ˆλ‹€.
139
+
140
+ ---
141
+
142
+ 규격 v1.1 (8개 μ‘°ν•­) Β· **FINAL-Bench / VIDRAFT**