Link the public Gaia benchmark notebooks

#1
by deeeech - opened
Files changed (1) hide show
  1. README.md +62 -1
README.md CHANGED
@@ -1,5 +1,66 @@
 
 
 
 
 
 
 
 
1
  # Gaia
2
 
3
- Gaia is a world model for solid tumor oncology.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
 
5
  Project overview: https://blog.alunadata.com/
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ tags:
4
+ - oncology
5
+ - spatial-biology
6
+ - benchmarks
7
+ ---
8
+
9
  # Gaia
10
 
11
+ Gaia is a world model for solid-tumor oncology.
12
+
13
+ ## Public benchmark notebooks
14
+
15
+ The public benchmark release is separated by evaluation scale. Each notebook
16
+ is fully executed, explains every reported score, and links its compact
17
+ evaluation script and input tables.
18
+
19
+ - **[Cohort-level benchmarks](https://github.com/alunalabs/gaia/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, direct predicted-versus-observed ORR comparisons, and controls.
20
+ - **[Patient-level benchmarks](https://github.com/alunalabs/gaia/blob/main/02_patient_level_benchmarks.ipynb):** the current CRC and cSCC patient-response scores, ROC AUC evaluation, exact label controls, and CRC measured on-treatment biology.
21
+ - **[Biology-level benchmarks](https://github.com/alunalabs/gaia/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row BioBench protocol, exact KO/OE scoring rules, row-family inventory, and KO gene-direction audit.
22
+
23
+ The complete benchmark package is in the
24
+ **[public Gaia GitHub repository](https://github.com/alunalabs/gaia)**.
25
+
26
+ ## Released results
27
+
28
+ | Evaluation | Result |
29
+ | --- | --- |
30
+ | Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
31
+ | CRC pretreatment response, 11 patients | ROC AUC 0.800 |
32
+ | cSCC pretreatment response, 12 patients | ROC AUC 0.778 |
33
+ | CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
34
+ | BioBench sender knockout | 257/289 direction passes (88.9%) |
35
+ | BioBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
36
+ | BioBench KO gene-direction audit | 0/173 audited ligand rows had every observed pathway gene decrease; median fraction decreasing 61.9% |
37
+
38
+ BioBench contains 183 ligand–receptor pathway rows and 106 regional
39
+ tissue-program rows. The separate per-gene audit is available for 173 ligand
40
+ rows. It rules out an all-pathway-genes-down explanation on those rows, not all
41
+ possible global-shift confounding; 10 ligand rows and the 106 regional rows do
42
+ not have the same archived gene-level audit.
43
+
44
+ ## Reproduction boundary
45
+
46
+ The GitHub release reproduces metrics, controls, tables, and figures from
47
+ compact exported benchmark data. A public model-adapter runner owns the cohort,
48
+ patient, and BioBench orchestration and score construction; the actual model,
49
+ weights, tissue loader, and encoder remain behind the adapter. The included
50
+ released-table adapter is explicitly a deterministic smoke test, not fresh
51
+ inference. The repository contains no training code.
52
+
53
+ This Hugging Face repository currently contains the project card only; model
54
+ weights are not currently published here.
55
+
56
+ The released benchmarks are research evaluations, not prospective clinical
57
+ validation or a clinical decision system.
58
+
59
+ ## License
60
+
61
+ Repository-authored release material is available under
62
+ [CC BY-NC 4.0](https://github.com/alunalabs/gaia/blob/main/LICENSE). Commercial
63
+ use is not permitted under that license. Underlying publications, study
64
+ records, databases, and source datasets retain their own terms.
65
 
66
  Project overview: https://blog.alunadata.com/