Link the public Gaia benchmark notebooks
#1
by deeeech - opened
README.md
CHANGED
|
@@ -1,5 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Gaia
|
| 2 |
|
| 3 |
-
Gaia is a world model for solid
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
Project overview: https://blog.alunadata.com/
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
tags:
|
| 4 |
+
- oncology
|
| 5 |
+
- spatial-biology
|
| 6 |
+
- benchmarks
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
# Gaia
|
| 10 |
|
| 11 |
+
Gaia is a world model for solid-tumor oncology.
|
| 12 |
+
|
| 13 |
+
## Public benchmark notebooks
|
| 14 |
+
|
| 15 |
+
The public benchmark release is separated by evaluation scale. Each notebook
|
| 16 |
+
is fully executed, explains every reported score, and links its compact
|
| 17 |
+
evaluation script and input tables.
|
| 18 |
+
|
| 19 |
+
- **[Cohort-level benchmarks](https://github.com/alunalabs/gaia/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, direct predicted-versus-observed ORR comparisons, and controls.
|
| 20 |
+
- **[Patient-level benchmarks](https://github.com/alunalabs/gaia/blob/main/02_patient_level_benchmarks.ipynb):** the current CRC and cSCC patient-response scores, ROC AUC evaluation, exact label controls, and CRC measured on-treatment biology.
|
| 21 |
+
- **[Biology-level benchmarks](https://github.com/alunalabs/gaia/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row BioBench protocol, exact KO/OE scoring rules, row-family inventory, and KO gene-direction audit.
|
| 22 |
+
|
| 23 |
+
The complete benchmark package is in the
|
| 24 |
+
**[public Gaia GitHub repository](https://github.com/alunalabs/gaia)**.
|
| 25 |
+
|
| 26 |
+
## Released results
|
| 27 |
+
|
| 28 |
+
| Evaluation | Result |
|
| 29 |
+
| --- | --- |
|
| 30 |
+
| Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
|
| 31 |
+
| CRC pretreatment response, 11 patients | ROC AUC 0.800 |
|
| 32 |
+
| cSCC pretreatment response, 12 patients | ROC AUC 0.778 |
|
| 33 |
+
| CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
|
| 34 |
+
| BioBench sender knockout | 257/289 direction passes (88.9%) |
|
| 35 |
+
| BioBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
|
| 36 |
+
| BioBench KO gene-direction audit | 0/173 audited ligand rows had every observed pathway gene decrease; median fraction decreasing 61.9% |
|
| 37 |
+
|
| 38 |
+
BioBench contains 183 ligand–receptor pathway rows and 106 regional
|
| 39 |
+
tissue-program rows. The separate per-gene audit is available for 173 ligand
|
| 40 |
+
rows. It rules out an all-pathway-genes-down explanation on those rows, not all
|
| 41 |
+
possible global-shift confounding; 10 ligand rows and the 106 regional rows do
|
| 42 |
+
not have the same archived gene-level audit.
|
| 43 |
+
|
| 44 |
+
## Reproduction boundary
|
| 45 |
+
|
| 46 |
+
The GitHub release reproduces metrics, controls, tables, and figures from
|
| 47 |
+
compact exported benchmark data. A public model-adapter runner owns the cohort,
|
| 48 |
+
patient, and BioBench orchestration and score construction; the actual model,
|
| 49 |
+
weights, tissue loader, and encoder remain behind the adapter. The included
|
| 50 |
+
released-table adapter is explicitly a deterministic smoke test, not fresh
|
| 51 |
+
inference. The repository contains no training code.
|
| 52 |
+
|
| 53 |
+
This Hugging Face repository currently contains the project card only; model
|
| 54 |
+
weights are not currently published here.
|
| 55 |
+
|
| 56 |
+
The released benchmarks are research evaluations, not prospective clinical
|
| 57 |
+
validation or a clinical decision system.
|
| 58 |
+
|
| 59 |
+
## License
|
| 60 |
+
|
| 61 |
+
Repository-authored release material is available under
|
| 62 |
+
[CC BY-NC 4.0](https://github.com/alunalabs/gaia/blob/main/LICENSE). Commercial
|
| 63 |
+
use is not permitted under that license. Underlying publications, study
|
| 64 |
+
records, databases, and source datasets retain their own terms.
|
| 65 |
|
| 66 |
Project overview: https://blog.alunadata.com/
|