Align card with ORR-only, AUC-only, and BioBench release
Browse files
README.md
CHANGED
|
@@ -16,61 +16,39 @@ The public benchmark release is separated by evaluation scale. Each notebook
|
|
| 16 |
is fully executed, explains every reported score, and links its compact
|
| 17 |
evaluation script and input tables.
|
| 18 |
|
| 19 |
-
- **[Cohort-level benchmarks](https://github.com/alunalabs/
|
| 20 |
-
- **[Patient-level benchmarks](https://github.com/alunalabs/
|
| 21 |
-
- **[Biology-level benchmarks](https://github.com/alunalabs/
|
| 22 |
|
| 23 |
The complete benchmark package is in the
|
| 24 |
-
**[public Gaia GitHub repository](https://github.com/alunalabs/
|
| 25 |
-
|
| 26 |
-
## Shared gene-readout efficacy score
|
| 27 |
-
|
| 28 |
-
The cohort, CRC, and cSCC evaluations use the same final arithmetic:
|
| 29 |
-
|
| 30 |
-
```text
|
| 31 |
-
bio_conversion_support = mean(response_conversion_support,
|
| 32 |
-
terminal_efficacy_support)
|
| 33 |
-
|
| 34 |
-
universal_gene_readout_score = coverage_support
|
| 35 |
-
* bio_conversion_support
|
| 36 |
-
* resistant_tail_control
|
| 37 |
-
```
|
| 38 |
-
|
| 39 |
-
The terminal readout is therapy-aware: cohort and CRC use apoptosis support;
|
| 40 |
-
cSCC checkpoint blockade uses control of persistent immune escape. The score
|
| 41 |
-
is unitless and is not a calibrated clinical response probability or ORR.
|
| 42 |
|
| 43 |
## Released results
|
| 44 |
|
| 45 |
| Evaluation | Result |
|
| 46 |
| --- | --- |
|
| 47 |
-
| Cohort
|
| 48 |
-
|
|
| 49 |
-
|
|
| 50 |
-
| cSCC shared-formula response, 12 patients | ROC AUC 0.778; Spearman 0.483; exact p=0.0660 |
|
| 51 |
-
| Previous cSCC-specific five-axis comparator, same patients | ROC AUC 0.944; Spearman 0.772 |
|
| 52 |
| CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
|
| 53 |
-
|
|
| 54 |
-
|
|
| 55 |
-
|
| 56 |
-
The previous cSCC-specific score is shown because it discriminates better, but
|
| 57 |
-
it is not the cross-benchmark headline. Enforcing the shared formula reduces
|
| 58 |
-
cSCC AUC from 0.944 to 0.778; this is the cost of comparability, not an
|
| 59 |
-
improvement. A strict literal-apoptosis cSCC audit reaches AUC 0.417 and is not
|
| 60 |
-
presented as a supported checkpoint terminal mechanism.
|
| 61 |
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
|
|
|
| 66 |
|
| 67 |
## Reproduction boundary
|
| 68 |
|
| 69 |
-
The GitHub release reproduces
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
|
|
|
| 74 |
|
| 75 |
This Hugging Face repository currently contains the project card only; model
|
| 76 |
weights are not currently published here.
|
|
@@ -81,7 +59,7 @@ validation or a clinical decision system.
|
|
| 81 |
## License
|
| 82 |
|
| 83 |
Repository-authored release material is available under
|
| 84 |
-
[CC BY-NC 4.0](https://github.com/alunalabs/
|
| 85 |
use is not permitted under that license. Underlying publications, study
|
| 86 |
records, databases, and source datasets retain their own terms.
|
| 87 |
|
|
|
|
| 16 |
is fully executed, explains every reported score, and links its compact
|
| 17 |
evaluation script and input tables.
|
| 18 |
|
| 19 |
+
- **[Cohort-level benchmarks](https://github.com/alunalabs/gaia/blob/main/01_cohort_level_benchmarks.ipynb):** drug perturbation priors, the complete 44-row trial-ORR source ledger, direct predicted-versus-observed ORR comparisons, and controls.
|
| 20 |
+
- **[Patient-level benchmarks](https://github.com/alunalabs/gaia/blob/main/02_patient_level_benchmarks.ipynb):** the current CRC and cSCC patient-response scores, ROC AUC evaluation, exact label controls, and CRC measured on-treatment biology.
|
| 21 |
+
- **[Biology-level benchmarks](https://github.com/alunalabs/gaia/blob/main/03_biology_level_benchmarks.ipynb):** the 289-row BioBench protocol, exact KO/OE scoring rules, row-family inventory, and KO gene-direction audit.
|
| 22 |
|
| 23 |
The complete benchmark package is in the
|
| 24 |
+
**[public Gaia GitHub repository](https://github.com/alunalabs/gaia)**.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
## Released results
|
| 27 |
|
| 28 |
| Evaluation | Result |
|
| 29 |
| --- | --- |
|
| 30 |
+
| Cohort ORR, 44 disease–drug rows | Pearson 0.650; Spearman 0.594; MAE 10.25 ORR percentage points |
|
| 31 |
+
| CRC pretreatment response, 11 patients | ROC AUC 0.800 |
|
| 32 |
+
| cSCC pretreatment response, 12 patients | ROC AUC 0.778 |
|
|
|
|
|
|
|
| 33 |
| CRC measured on-treatment biology, 11 patients | Best descriptive mean cosine 0.304 at rollout step 4 |
|
| 34 |
+
| BioBench sender knockout | 257/289 direction passes (88.9%) |
|
| 35 |
+
| BioBench sender overexpression | 257/281 scoreable direction passes (91.5%); 8 rows not applicable |
|
| 36 |
+
| BioBench KO gene-direction audit | 0/173 audited ligand rows had every observed pathway gene decrease; median fraction decreasing 61.9% |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
+
BioBench contains 183 ligand–receptor pathway rows and 106 regional
|
| 39 |
+
tissue-program rows. The separate per-gene audit is available for 173 ligand
|
| 40 |
+
rows. It rules out an all-pathway-genes-down explanation on those rows, not all
|
| 41 |
+
possible global-shift confounding; 10 ligand rows and the 106 regional rows do
|
| 42 |
+
not have the same archived gene-level audit.
|
| 43 |
|
| 44 |
## Reproduction boundary
|
| 45 |
|
| 46 |
+
The GitHub release reproduces metrics, controls, tables, and figures from
|
| 47 |
+
compact exported benchmark data. A public model-adapter runner owns the cohort,
|
| 48 |
+
patient, and BioBench orchestration and score construction; the actual model,
|
| 49 |
+
weights, tissue loader, and encoder remain behind the adapter. The included
|
| 50 |
+
released-table adapter is explicitly a deterministic smoke test, not fresh
|
| 51 |
+
inference. The repository contains no training code.
|
| 52 |
|
| 53 |
This Hugging Face repository currently contains the project card only; model
|
| 54 |
weights are not currently published here.
|
|
|
|
| 59 |
## License
|
| 60 |
|
| 61 |
Repository-authored release material is available under
|
| 62 |
+
[CC BY-NC 4.0](https://github.com/alunalabs/gaia/blob/main/LICENSE). Commercial
|
| 63 |
use is not permitted under that license. Underlying publications, study
|
| 64 |
records, databases, and source datasets retain their own terms.
|
| 65 |
|