Update benchmark links and released results
Browse files
README.md
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
---
|
|
|
|
| 2 |
tags:
|
| 3 |
- oncology
|
| 4 |
- spatial-biology
|
|
@@ -9,22 +10,54 @@ tags:
|
|
| 9 |
|
| 10 |
Gaia is a world model for solid-tumor oncology.
|
| 11 |
|
| 12 |
-
## Public benchmark
|
| 13 |
|
| 14 |
-
The public release
|
|
|
|
|
|
|
| 15 |
|
| 16 |
-
- **Cohort-level:**
|
| 17 |
-
- **Patient-level:**
|
| 18 |
-
- **Biology-level:**
|
| 19 |
|
| 20 |
-
The
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
## Reproduction boundary
|
| 27 |
|
| 28 |
-
The
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
-
|
|
|
|
| 1 |
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
tags:
|
| 4 |
- oncology
|
| 5 |
- spatial-biology
|
|
|
|
| 10 |
|
| 11 |
Gaia is a world model for solid-tumor oncology.
|
| 12 |
|
| 13 |
+
## Public benchmark notebooks
|
| 14 |
|
| 15 |
+
The public benchmark release is separated by evaluation scale. Each notebook
|
| 16 |
+
is fully executed, explains every reported score, and links its compact
|
| 17 |
+
evaluation script and input tables.
|
| 18 |
|
| 19 |
+
- **[Cohort-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, observed-versus-predicted ORR correlations, comparisons, and permutation controls.
|
| 20 |
+
- **[Patient-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/02_patient_level_benchmarks.ipynb):** CRC and cSCC patient-response scores, complete-vector controls, and CRC measured on-treatment biology.
|
| 21 |
+
- **[Biology-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row CellChatBench protocol, row inventories, exact scoring rules, worked examples, and results.
|
| 22 |
|
| 23 |
+
The complete benchmark package is in the
|
| 24 |
+
**[public Gaia GitHub repository](https://github.com/alunalabs/GAIA)**.
|
| 25 |
|
| 26 |
+
## Released results
|
| 27 |
|
| 28 |
+
| Evaluation | Result |
|
| 29 |
+
| --- | --- |
|
| 30 |
+
| Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
|
| 31 |
+
| CRC pretreatment response, 11 patients | ROC AUC 0.800; balanced accuracy 0.733 |
|
| 32 |
+
| cSCC pretreatment response, 12 patients | ROC AUC 0.944; Spearman 0.772 |
|
| 33 |
+
| CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
|
| 34 |
+
| CellChatBench sender knockout | 257/289 direction passes (88.9%) |
|
| 35 |
+
| CellChatBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
|
| 36 |
+
|
| 37 |
+
The CellChatBench release contains 183 ligand–receptor pathway rows and 106
|
| 38 |
+
regional tissue-program rows. All released knockout statements expect a
|
| 39 |
+
receiver decrease and all overexpression statements expect an increase, so
|
| 40 |
+
these results do not establish inhibitory-sign balance.
|
| 41 |
|
| 42 |
## Reproduction boundary
|
| 43 |
|
| 44 |
+
The GitHub release reproduces metrics, controls, tables, and figures from
|
| 45 |
+
compact exported benchmark data. It contains evaluation scripts only—no model
|
| 46 |
+
training code—and does not rerun upstream spatial inference. Raw tissue assays,
|
| 47 |
+
source perturbation matrices, full per-cell outputs, checkpoints, and model
|
| 48 |
+
weights are not included.
|
| 49 |
+
|
| 50 |
+
This Hugging Face repository currently contains the project card only; model
|
| 51 |
+
weights are not currently published here.
|
| 52 |
+
|
| 53 |
+
The released benchmarks are research evaluations, not prospective clinical
|
| 54 |
+
validation or a clinical decision system.
|
| 55 |
+
|
| 56 |
+
## License
|
| 57 |
+
|
| 58 |
+
Repository-authored release material is available under
|
| 59 |
+
[CC BY-NC 4.0](https://github.com/alunalabs/GAIA/blob/main/LICENSE). Commercial
|
| 60 |
+
use is not permitted under that license. Underlying publications, study
|
| 61 |
+
records, databases, and source datasets retain their own terms.
|
| 62 |
|
| 63 |
+
Project overview: https://blog.alunadata.com/
|