deeeech commited on
Commit
a12b8dc
·
verified ·
1 Parent(s): 7f9cdba

Align card with ORR-only, AUC-only, and BioBench release

Browse files
Files changed (1) hide show
  1. README.md +22 -44
README.md CHANGED
@@ -16,61 +16,39 @@ The public benchmark release is separated by evaluation scale. Each notebook
16
  is fully executed, explains every reported score, and links its compact
17
  evaluation script and input tables.
18
 
19
- - **[Cohort-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the shared gene-readout efficacy score, all 44 trial ORRs and their source ledger, a separate ORR-calibrated output, comparisons, and controls.
20
- - **[Patient-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/02_patient_level_benchmarks.ipynb):** the same four-gate score applied to CRC and cSCC, exact label controls, cSCC terminal-readout audits, and CRC measured on-treatment biology.
21
- - **[Biology-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row CellChatBench protocol, row inventories, exact scoring rules, worked examples, and results.
22
 
23
  The complete benchmark package is in the
24
- **[public Gaia GitHub repository](https://github.com/alunalabs/GAIA)**.
25
-
26
- ## Shared gene-readout efficacy score
27
-
28
- The cohort, CRC, and cSCC evaluations use the same final arithmetic:
29
-
30
- ```text
31
- bio_conversion_support = mean(response_conversion_support,
32
- terminal_efficacy_support)
33
-
34
- universal_gene_readout_score = coverage_support
35
- * bio_conversion_support
36
- * resistant_tail_control
37
- ```
38
-
39
- The terminal readout is therapy-aware: cohort and CRC use apoptosis support;
40
- cSCC checkpoint blockade uses control of persistent immune escape. The score
41
- is unitless and is not a calibrated clinical response probability or ORR.
42
 
43
  ## Released results
44
 
45
  | Evaluation | Result |
46
  | --- | --- |
47
- | Cohort universal gene-readout efficacy, 44 disease–drug rows | Pearson 0.658; Spearman 0.626 |
48
- | Separate cohort ORR-calibrated output, same 44 rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
49
- | CRC shared-formula response, 11 patients | ROC AUC 0.800; Spearman 0.520; exact p=0.0628 |
50
- | cSCC shared-formula response, 12 patients | ROC AUC 0.778; Spearman 0.483; exact p=0.0660 |
51
- | Previous cSCC-specific five-axis comparator, same patients | ROC AUC 0.944; Spearman 0.772 |
52
  | CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
53
- | CellChatBench sender knockout | 257/289 direction passes (88.9%) |
54
- | CellChatBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
55
-
56
- The previous cSCC-specific score is shown because it discriminates better, but
57
- it is not the cross-benchmark headline. Enforcing the shared formula reduces
58
- cSCC AUC from 0.944 to 0.778; this is the cost of comparability, not an
59
- improvement. A strict literal-apoptosis cSCC audit reaches AUC 0.417 and is not
60
- presented as a supported checkpoint terminal mechanism.
61
 
62
- The CellChatBench release contains 183 ligand–receptor pathway rows and 106
63
- regional tissue-program rows. All released knockout statements expect a
64
- receiver decrease and all overexpression statements expect an increase, so
65
- these results do not establish inhibitory-sign balance.
 
66
 
67
  ## Reproduction boundary
68
 
69
- The GitHub release reproduces formulas, aggregation, metrics, controls, tables,
70
- and figures from compact exported benchmark data. It contains evaluation
71
- scripts only—no model training code—and does not rerun upstream spatial
72
- inference. Raw tissue assays, source perturbation matrices, full per-cell
73
- outputs, checkpoints, and model weights are not included.
 
74
 
75
  This Hugging Face repository currently contains the project card only; model
76
  weights are not currently published here.
@@ -81,7 +59,7 @@ validation or a clinical decision system.
81
  ## License
82
 
83
  Repository-authored release material is available under
84
- [CC BY-NC 4.0](https://github.com/alunalabs/GAIA/blob/main/LICENSE). Commercial
85
  use is not permitted under that license. Underlying publications, study
86
  records, databases, and source datasets retain their own terms.
87
 
 
16
  is fully executed, explains every reported score, and links its compact
17
  evaluation script and input tables.
18
 
19
+ - **[Cohort-level benchmarks](https://github.com/alunalabs/gaia/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, direct predicted-versus-observed ORR comparisons, and controls.
20
+ - **[Patient-level benchmarks](https://github.com/alunalabs/gaia/blob/main/02_patient_level_benchmarks.ipynb):** the current CRC and cSCC patient-response scores, ROC AUC evaluation, exact label controls, and CRC measured on-treatment biology.
21
+ - **[Biology-level benchmarks](https://github.com/alunalabs/gaia/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row BioBench protocol, exact KO/OE scoring rules, row-family inventory, and KO gene-direction audit.
22
 
23
  The complete benchmark package is in the
24
+ **[public Gaia GitHub repository](https://github.com/alunalabs/gaia)**.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
 
26
  ## Released results
27
 
28
  | Evaluation | Result |
29
  | --- | --- |
30
+ | Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
31
+ | CRC pretreatment response, 11 patients | ROC AUC 0.800 |
32
+ | cSCC pretreatment response, 12 patients | ROC AUC 0.778 |
 
 
33
  | CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
34
+ | BioBench sender knockout | 257/289 direction passes (88.9%) |
35
+ | BioBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
36
+ | BioBench KO gene-direction audit | 0/173 audited ligand rows had every observed pathway gene decrease; median fraction decreasing 61.9% |
 
 
 
 
 
37
 
38
+ BioBench contains 183 ligand–receptor pathway rows and 106 regional
39
+ tissue-program rows. The separate per-gene audit is available for 173 ligand
40
+ rows. It rules out an all-pathway-genes-down explanation on those rows, not all
41
+ possible global-shift confounding; 10 ligand rows and the 106 regional rows do
42
+ not have the same archived gene-level audit.
43
 
44
  ## Reproduction boundary
45
 
46
+ The GitHub release reproduces metrics, controls, tables, and figures from
47
+ compact exported benchmark data. A public model-adapter runner owns the cohort,
48
+ patient, and BioBench orchestration and score construction; the actual model,
49
+ weights, tissue loader, and encoder remain behind the adapter. The included
50
+ released-table adapter is explicitly a deterministic smoke test, not fresh
51
+ inference. The repository contains no training code.
52
 
53
  This Hugging Face repository currently contains the project card only; model
54
  weights are not currently published here.
 
59
  ## License
60
 
61
  Repository-authored release material is available under
62
+ [CC BY-NC 4.0](https://github.com/alunalabs/gaia/blob/main/LICENSE). Commercial
63
  use is not permitted under that license. Underlying publications, study
64
  records, databases, and source datasets retain their own terms.
65