deeeech commited on
Commit
7f9cdba
·
verified ·
1 Parent(s): c703adb

Document shared cohort CRC and cSCC efficacy score

Browse files
Files changed (1) hide show
  1. README.md +35 -10
README.md CHANGED
@@ -16,24 +16,49 @@ The public benchmark release is separated by evaluation scale. Each notebook
16
  is fully executed, explains every reported score, and links its compact
17
  evaluation script and input tables.
18
 
19
- - **[Cohort-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, observed-versus-predicted ORR correlations, comparisons, and permutation controls.
20
- - **[Patient-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/02_patient_level_benchmarks.ipynb):** CRC and cSCC patient-response scores, complete-vector controls, and CRC measured on-treatment biology.
21
  - **[Biology-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row CellChatBench protocol, row inventories, exact scoring rules, worked examples, and results.
22
 
23
  The complete benchmark package is in the
24
  **[public Gaia GitHub repository](https://github.com/alunalabs/GAIA)**.
25
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ## Released results
27
 
28
  | Evaluation | Result |
29
  | --- | --- |
30
- | Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
31
- | CRC pretreatment response, 11 patients | ROC AUC 0.800; balanced accuracy 0.733 |
32
- | cSCC pretreatment response, 12 patients | ROC AUC 0.944; Spearman 0.772 |
 
 
33
  | CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
34
  | CellChatBench sender knockout | 257/289 direction passes (88.9%) |
35
  | CellChatBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
36
 
 
 
 
 
 
 
37
  The CellChatBench release contains 183 ligand–receptor pathway rows and 106
38
  regional tissue-program rows. All released knockout statements expect a
39
  receiver decrease and all overexpression statements expect an increase, so
@@ -41,11 +66,11 @@ these results do not establish inhibitory-sign balance.
41
 
42
  ## Reproduction boundary
43
 
44
- The GitHub release reproduces metrics, controls, tables, and figures from
45
- compact exported benchmark data. It contains evaluation scripts only—no model
46
- training code—and does not rerun upstream spatial inference. Raw tissue assays,
47
- source perturbation matrices, full per-cell outputs, checkpoints, and model
48
- weights are not included.
49
 
50
  This Hugging Face repository currently contains the project card only; model
51
  weights are not currently published here.
 
16
  is fully executed, explains every reported score, and links its compact
17
  evaluation script and input tables.
18
 
19
+ - **[Cohort-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the shared gene-readout efficacy score, all 44 trial ORRs and their source ledger, a separate ORR-calibrated output, comparisons, and controls.
20
+ - **[Patient-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/02_patient_level_benchmarks.ipynb):** the same four-gate score applied to CRC and cSCC, exact label controls, cSCC terminal-readout audits, and CRC measured on-treatment biology.
21
  - **[Biology-level benchmarks](https://github.com/alunalabs/GAIA/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row CellChatBench protocol, row inventories, exact scoring rules, worked examples, and results.
22
 
23
  The complete benchmark package is in the
24
  **[public Gaia GitHub repository](https://github.com/alunalabs/GAIA)**.
25
 
26
+ ## Shared gene-readout efficacy score
27
+
28
+ The cohort, CRC, and cSCC evaluations use the same final arithmetic:
29
+
30
+ ```text
31
+ bio_conversion_support = mean(response_conversion_support,
32
+ terminal_efficacy_support)
33
+
34
+ universal_gene_readout_score = coverage_support
35
+ * bio_conversion_support
36
+ * resistant_tail_control
37
+ ```
38
+
39
+ The terminal readout is therapy-aware: cohort and CRC use apoptosis support;
40
+ cSCC checkpoint blockade uses control of persistent immune escape. The score
41
+ is unitless and is not a calibrated clinical response probability or ORR.
42
+
43
  ## Released results
44
 
45
  | Evaluation | Result |
46
  | --- | --- |
47
+ | Cohort universal gene-readout efficacy, 44 disease–drug rows | Pearson 0.658; Spearman 0.626 |
48
+ | Separate cohort ORR-calibrated output, same 44 rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
49
+ | CRC shared-formula response, 11 patients | ROC AUC 0.800; Spearman 0.520; exact p=0.0628 |
50
+ | cSCC shared-formula response, 12 patients | ROC AUC 0.778; Spearman 0.483; exact p=0.0660 |
51
+ | Previous cSCC-specific five-axis comparator, same patients | ROC AUC 0.944; Spearman 0.772 |
52
  | CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
53
  | CellChatBench sender knockout | 257/289 direction passes (88.9%) |
54
  | CellChatBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
55
 
56
+ The previous cSCC-specific score is shown because it discriminates better, but
57
+ it is not the cross-benchmark headline. Enforcing the shared formula reduces
58
+ cSCC AUC from 0.944 to 0.778; this is the cost of comparability, not an
59
+ improvement. A strict literal-apoptosis cSCC audit reaches AUC 0.417 and is not
60
+ presented as a supported checkpoint terminal mechanism.
61
+
62
  The CellChatBench release contains 183 ligand–receptor pathway rows and 106
63
  regional tissue-program rows. All released knockout statements expect a
64
  receiver decrease and all overexpression statements expect an increase, so
 
66
 
67
  ## Reproduction boundary
68
 
69
+ The GitHub release reproduces formulas, aggregation, metrics, controls, tables,
70
+ and figures from compact exported benchmark data. It contains evaluation
71
+ scripts only—no model training code—and does not rerun upstream spatial
72
+ inference. Raw tissue assays, source perturbation matrices, full per-cell
73
+ outputs, checkpoints, and model weights are not included.
74
 
75
  This Hugging Face repository currently contains the project card only; model
76
  weights are not currently published here.