deeeech commited on
Commit
c703adb
·
verified ·
1 Parent(s): 2d7a357

Update benchmark links and released results

Browse files
Files changed (1) hide show
  1. README.md +43 -10
README.md CHANGED
@@ -1,4 +1,5 @@
1
  ---
 
2
  tags:
3
  - oncology
4
  - spatial-biology
@@ -9,22 +10,54 @@ tags:
9
 
10
  Gaia is a world model for solid-tumor oncology.
11
 
12
- ## Public benchmark report
13
 
14
- The public release reports three complementary evaluation levels:
 
 
15
 
16
- - **Cohort-level:** observed versus predicted objective response rate across 44 disease–treatment cohorts.
17
- - **Patient-level:** response ranking in colorectal cancer and cutaneous squamous cell carcinoma, plus a measured on-treatment biology analysis.
18
- - **Biology-level:** cell–cell communication relationship recovery across 2,614 requested biological contexts.
19
 
20
- The fully executed notebook is the main report. It defines every score, shows the underlying tables, and includes all correlation, control, patient-level, and biology-level plots:
 
21
 
22
- **[Open the Gaia benchmark notebook](https://github.com/alunalabs/GAIA/blob/main/Gaia_benchmarks.ipynb)**
23
 
24
- The compact benchmark scripts and public input tables are available in the [Gaia GitHub repository](https://github.com/alunalabs/GAIA).
 
 
 
 
 
 
 
 
 
 
 
 
25
 
26
  ## Reproduction boundary
27
 
28
- The public release reproduces benchmark metrics and figures from the included evaluation tables. It does not rerun model inference or model training. This Hugging Face repository does not currently contain model weights.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
- The materials are intended for research use and are not a clinical decision system.
 
1
  ---
2
+ license: cc-by-nc-4.0
3
  tags:
4
  - oncology
5
  - spatial-biology
 
10
 
11
  Gaia is a world model for solid-tumor oncology.
12
 
13
+ ## Public benchmark notebooks
14
 
15
+ The public benchmark release is separated by evaluation scale. Each notebook
16
+ is fully executed, explains every reported score, and links its compact
17
+ evaluation script and input tables.
18
 
19
+ - **[Cohort-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, observed-versus-predicted ORR correlations, comparisons, and permutation controls.
20
+ - **[Patient-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/02_patient_level_benchmarks.ipynb):** CRC and cSCC patient-response scores, complete-vector controls, and CRC measured on-treatment biology.
21
+ - **[Biology-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row CellChatBench protocol, row inventories, exact scoring rules, worked examples, and results.
22
 
23
+ The complete benchmark package is in the
24
+ **[public Gaia GitHub repository](https://github.com/alunalabs/GAIA)**.
25
 
26
+ ## Released results
27
 
28
+ | Evaluation | Result |
29
+ | --- | --- |
30
+ | Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
31
+ | CRC pretreatment response, 11 patients | ROC AUC 0.800; balanced accuracy 0.733 |
32
+ | cSCC pretreatment response, 12 patients | ROC AUC 0.944; Spearman 0.772 |
33
+ | CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
34
+ | CellChatBench sender knockout | 257/289 direction passes (88.9%) |
35
+ | CellChatBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
36
+
37
+ The CellChatBench release contains 183 ligand–receptor pathway rows and 106
38
+ regional tissue-program rows. All released knockout statements expect a
39
+ receiver decrease and all overexpression statements expect an increase, so
40
+ these results do not establish inhibitory-sign balance.
41
 
42
  ## Reproduction boundary
43
 
44
+ The GitHub release reproduces metrics, controls, tables, and figures from
45
+ compact exported benchmark data. It contains evaluation scripts only—no model
46
+ training code—and does not rerun upstream spatial inference. Raw tissue assays,
47
+ source perturbation matrices, full per-cell outputs, checkpoints, and model
48
+ weights are not included.
49
+
50
+ This Hugging Face repository currently contains the project card only; model
51
+ weights are not currently published here.
52
+
53
+ The released benchmarks are research evaluations, not prospective clinical
54
+ validation or a clinical decision system.
55
+
56
+ ## License
57
+
58
+ Repository-authored release material is available under
59
+ [CC BY-NC 4.0](https://github.com/alunalabs/GAIA/blob/main/LICENSE). Commercial
60
+ use is not permitted under that license. Underlying publications, study
61
+ records, databases, and source datasets retain their own terms.
62
 
63
+ Project overview: https://blog.alunadata.com/