DATA.md

# Datasets, embeddings and studies

Everything in this repository, what it is, and what it is for. Counts were read back from the
Hugging Face API after upload.

| Directory | Contents |
|---|---|
| [`graphs/`](#graphs) | cell graphs, the model's inputs |
| [`embeddings/`](#embeddings) | embeddings produced by `graphist_v2.pt` |
| [`baselines/`](#baselines) | DINOv2, MAE and GrapHist v1 features for comparison |
| [`labels/`](#labels) | TCGA-BRCA clinical table and slide labels |
| [`studies/`](#studies) | side analyses, each self-contained |
| [`upstream_v1/`](#upstream_v1) | mirror of the original GrapHist v1 releases |

---

## ⚠️ Read this before using `graphs/`

The label CSVs store **bare filenames** in their `graph_path` column, and the loader passes
that column to `torch.load` unchanged, so it resolves against the **process working directory**
rather than against the CSV. So unless you happen to run from inside the graph folder, every path
fails, and the failure is quiet: unreadable paths are dropped with a single
`WARNING: Filtered out <n> missing/empty graph files`, leaving you with a *smaller* dataset,
or an empty one, which surfaces later as an unhelpful `IndexError`.

Fix it once per dataset, after extracting, with the helper in `modeling/`:

```bash
python modeling/rebase_graph_paths.py --csv <path to sample_labels_rich.csv> \
       --mode absolute --root "$PWD/<path to the .pt files>"
```

It rewrites by basename, so it does not matter where the graphs ended up, and it is
idempotent. Verify before training:

```bash
python modeling/rebase_graph_paths.py --csv <csv> --check     # expect N/N
```

Expected: BACH 14,341, BreakHis 709, BRACS 96,153, SPIDER-breast 71,745,
TCGA-BRCA 11,149,499. Anything less means graphs are being silently skipped.

## Graph format

| Tensor | Shape | Meaning |
|---|---|---|
| `x` | `[num_nodes, 96]` | per-cell morphology, texture and colour features |
| `edge_index` | `[2, num_edges]` | Delaunay edges, pruned at 100 µm |
| `edge_attr` | `[num_edges, 75]` | column 0 = centroid distance (µm); columns 1–74 describe the inter-cellular region |

Every dataset ships the same four companion files, and all four are needed.
`.pt` graphs alone are not enough:

| File | Purpose |
|---|---|
| `sample_labels_rich.csv` | graph → sample id → label (the file you rebase) |
| `sample_split.csv` | train / test assignment |
| `sample_labels.csv` | slide-level labels |
| `normalization.json` | per-feature mean/std for `NormalizeData` |

---

## graphs/

Large datasets ship as tars because Hugging Face documents a ceiling of 10,000 entries per folder.

### `graphs/tcga_brca/`: 11,149,499 graphs, 652 GB
The **pre-training corpus**. 324 × ~2.0 GB tars under `tcga_brca/data/`, plus the four
companion files (its `sample_labels_rich.csv` is 1.94 GB). Also the in-domain cohort for
slide-level evaluation.

Run from the repository root:

```bash
mkdir -p graphs/tcga_brca/extracted
for t in graphs/tcga_brca/data/tcga-brca-*.tar; do tar -xf "$t" -C graphs/tcga_brca/extracted; done
python modeling/rebase_graph_paths.py --csv graphs/tcga_brca/sample_labels_rich.csv \
       --mode absolute --root "$PWD/graphs/tcga_brca/extracted"
```

### `graphs/bach/`: 14,341 graphs
BACH (ICIAR 2018), 224 px @ 20×, 4-class. One `graphs.tar` (flat: the `.pt` files **and**
`sample_labels_rich.csv`) plus the three other companion files. Extract into a directory you
create yourself:

```bash
mkdir -p graphs/bach/extracted && tar -xf graphs/bach/graphs.tar -C graphs/bach/extracted
python modeling/rebase_graph_paths.py --csv graphs/bach/extracted/sample_labels_rich.csv \
       --mode absolute --root "$PWD/graphs/bach/extracted"
```

### `graphs/bracs/`: 96,153 graphs
BRACS, 7-class lesion subtyping. 6 × ~1 GB tars under `bracs/data/` holding **only** `.pt`
files. Unlike BACH, the `sample_labels_rich.csv` sits one level up at `bracs/`.

### `graphs/breakhis/`: 709 graphs
BreakHis, binary benign/malignant. Small enough to ship as loose `.pt` files under
`breakhis/graphs/`, with `sample_labels_rich.csv` alongside them.

### `graphs/spider_breast/`: 71,745 graphs
SPIDER-breast, 18 tumour-subtype classes, train 62,995 / test 8,750. 18 × ~245 MB tars under
`spider_breast/data/` (`.pt` only; the CSV is one level up).

Two things to know:

- **The label files cover 92,892 patches, the graphs are 71,745.** Graph construction skips
  any patch with fewer than 10 detected cells, which removes cell-sparse tissue almost
  entirely: `Fat` retains 7 of 6,286 patches (1 in test), `Fibrosis` 415 of 6,260,
  `Necrosis` 2,147 of 5,396, while ten other classes lose ~0 %. In an 18-class macro-F1 this
  matters, because `Fat` rests on a single test patch. `metadata.csv` is the file that matches the
  graphs exactly.
- **`sample_id` is doubled in `metadata.csv`** (`patch_0000017_patch_0000017`, matching the
  `.pt` filename) but single elsewhere. A naive join between them matches 0 rows; strip the
  doubling first. Do not "normalise" `metadata.csv`: the doubled form is what the embedding
  filenames need.

---

## embeddings/

Produced by `graphist_v2.pt`. Use these to skip inference entirely.

### `embeddings/slide/`: 6,531 slides
Per-slide `embeddings.h5`, dataset key `embeddings`, shape `(n_tiles, 512)`:
BACH 397, BRACS 4,493, BreakHis 522, TCGA-BRCA 1,119.

Feeds slide-level MIL subtyping and the TCGA-BRCA survival analysis (tiles → slide → patient
mean-pool, then Cox proportional hazards).

Each cohort also carries one `train/normalizer_values.json`. **Keep it.** The loader resolves
`<data>/../train/normalizer_values.json`, and if it is missing it silently recomputes the
statistics from whatever split you point at *and writes the file*, which changes your numbers
and requires a writable directory.

### `embeddings/cell/`: 22 tars
Per-graph `.npz` (keys `embedding` `(n, 512)`, `label` `(n,)`) for cell-type identification:
`NuCLS_{main,super}_fold1-5`, `PanNuke_{20x,40x}_test1-3`, `PanNuke_breast_{20x,40x}_test1-3`.
Evaluated with StandardScaler + multinomial logistic regression (`class_weight=balanced`,
`max_iter=2000`, seed 0).

These are the **only** route to the cell-level results: 75-dim NuCLS/PanNuke graphs were never
built, and the 1-dim v1 graphs are not a substitute (the encoder slices `edge_attr[:, 1:]` into
a `Linear(74, 512)`, so a 1-dim edge attribute is a shape error).

One known gap: every NuCLS tar holds 1,693 `.npz` against 1,694 graphs, and the same single
graph is absent from all ten. It is
`TCGA-S3-AA15-DX1_id-5ea40a6addda5f839898f24a_left-57268_top-29680_bottom-29958_right-57547`.
Effect on macro-F1 ≈ 0.06 %.

---

## baselines/

Tile features from other encoders on TCGA-BRCA, for the comparison rows in survival analysis
and batch-effect probing. Same `.h5` layout as `embeddings/slide/`.

| Directory | Model | Slides |
|---|---|---|
| `baselines/dinov2/` | DINOv2 ViT-S/14 | 1,126 |
| `baselines/mae/` | MAE ViT-S | 1,126 |
| `baselines/graphist_v1/embeddings/` | GrapHist v1 | 1,122 |

`baselines/graphist_v1/graphist_v1.pt` is the vanilla GrapHist v1 checkpoint (ACM-GIN, 1-dim
edges, 114,203,790 B). It is the fine-tuning starting point for the AdapterGNN study and the
encoder behind the v1 comparison numbers.

Slide counts differ slightly (1,126 / 1,122 / 1,119 for GrapHist++) because the embedding sets cover
marginally different slides. The survival scripts join on patient id, so each comparison runs
over its own intersection.

---

## labels/

| File | Contents |
|---|---|
| `tcga_brca_clinical.tsv` | GDC open-access clinical export, 5,546 rows over 1,098 patients |
| `tcga_brca_slide_labels.csv` | slide → label, header exactly `sample_id,label` |

The clinical table supplies the survival `time` / `event` columns; without it no survival
number can be computed. Four columns are read: `cases.submitter_id`,
`demographic.vital_status`, `demographic.days_to_death`,
`diagnoses.days_to_last_follow_up`.

There is **no site/hospital column**. The batch-effect analysis derives the hospital from
field 1 of the TCGA barcode.

Note on joins: for BACH, BreakHis and BRACS the label file is a superset of the split file
(every split id has a label). **TCGA-BRCA is the exception**: 1,122 labels against 1,126 split
rows, so four split ids have no label row and an inner join silently drops them.

---

## studies/

Side analyses. Each has its own folder here holding the data, the code that produced it
and its outputs, except the batch-effect study, which reuses `baselines/` and `labels/`
and so ships no folder of its own.

### `studies/homophily/`: how homophilic are cell graphs?

`H_node` is the fraction of a cell's neighbours sharing its label. The **fat tail** is the
share of cells below a threshold, i.e. cells whose neighbourhood is label-mixed, which a
homophily-assuming GNN will smooth incorrectly.

Per-node tails over the pooled graph, isolated nodes excluded:

| Cohort | labels | nodes | `H_node` < 0.3 | < 0.5 | pooled `H_edge` |
|---|---|--:|--:|--:|--:|
| NuCLS-super | ground truth, 4-class | 52,017 | 5.55 % | 10.77 % | 0.8287 |
| NuCLS-main | ground truth, 7-class | 52,017 | 7.93 % | 14.32 % | 0.7931 |
| PanNuke 20× | ground truth, 5-class | 162,241 | 10.46 % | 16.83 % | 0.7590 |
| PanNuke 40× | ground truth, 5-class | 162,241 | 10.48 % | 16.83 % | 0.7589 |
| BACH | CellViT++ pseudo, 4-class | 439,281 | 17.79 % | 28.62 % | 0.6304 |
| BRACS | CellViT++ pseudo, 4-class | 4,301,987 | 14.56 % | 23.31 % | 0.6917 |
| BreakHis | CellViT++ pseudo, 4-class | 6,255 | 3.60 % | 4.73 % | 0.9299 |

The two views disagree, and that is the finding: pooled edge homophily is **0.76–0.83** on the
labelled cohorts, so the graphs are *globally* homophilic, yet 5.6–10.5 % of individual cells
sit below `H_node` 0.3. The mean hides a heterophilic minority. Label granularity drives part
of it (NuCLS 4-class 5.55 % → 7-class 7.93 % on the *same* graphs). The per-*graph* tail is
below 1 % everywhere, so this is not a handful of pathological graphs.

`H_edge` above is **pooled** (micro-averaged over the disjoint union), not the per-graph
macro-mean, which runs 0.75–0.82 on the labelled cohorts.

Contents: `tiles/` holds 109,904 224 px tile graphs with CellViT++ pseudo-labels (BACH 13,837,
BRACS 95,670, BreakHis 397); `bach_cellvit/` holds 400 per-image pseudo-label CSVs (538,421
cells, columns `centroid_x_px,centroid_y_px,cell_type,type_prob`) plus a summary JSON; and
`paper_metrics/` holds the pooled edge-homophily and pseudo-label-gate results as JSON.

The pseudo-labeller was validated on a held-out ground-truth split first: **ARI 0.560**
(threshold 0.5), Cohen κ 0.709, per-class confusion diagonal 0.864 / 0.562 /
0.900 / 0.000. Three of four classes pass, and the fourth (`other_nucleus`) is untestable at
7 ground-truth instances. Re-running that gate needs the external CellViT++ repository and
SAM-H weights, which are not included here.

One caveat if you re-run the analysis: `_strip_virtual_node` removes the synthetic virtual
node by degree (`deg >= 0.95*(n-1)`), which on small graphs also deletes **real cells**, up to
676 nodes in a cohort, without warning. It shifts the tails by at most 0.037 pp, so no
conclusion changes, but it needs a minimum-degree guard.

### `studies/adaptergnn/`: parameter-efficient fine-tuning

Adapters inserted into a **frozen GrapHist v1** encoder for cell-type identification. Test
macro-F1 %, each the mean over cross-validation folds (2 to 5 folds depending on the column):

| Model | P20 Breast | P20 PanCancer | P40 Breast | P40 PanCancer | NuCLS main | NuCLS super |
|---|:--:|:--:|:--:|:--:|:--:|:--:|
| Supervised graph, ACM-bio | 56.61 | 62.73 | 57.06 | 65.80 | 22.13 | 37.35 |
| Supervised graph, ACM-UNI | 58.05 | **67.63** | 56.81 | 66.99 | 21.68 | 39.46 |
| DINOv2 probe | 54.82 | 50.49 | 53.86 | 49.27 | 21.42 | 41.17 |
| MAE probe | 47.71 | 54.88 | 47.54 | 55.26 | 25.19 | 45.31 |
| GrapHist v1 frozen probe | 55.26 | 58.78 | 56.43 | 59.47 | 26.57 | 46.24 |
| GrapHist v1 + AdapterGNN (shared recipe) | 57.54 | 65.86 | 57.07 | 64.92 | 27.22 | 44.47 |
| **GrapHist v1 + AdapterGNN (tuned)** | **58.42** | 66.79 | **59.19** | **70.93** | **27.55** | **48.65** |

The tuned protocol takes 5 of 6 columns. Against the frozen probe, which uses the same encoder,
so the comparison isolates the adapters, it gains +3.2 / +8.0 / +2.8 / +11.5 / +1.0 / +2.4.

Contents: `graphs_v1/` holds NuCLS (1,694 `.pt` × 2 label granularities plus 5-fold split
tables) and PanNuke (3 folds × 20×/40×, 14,414 `.pt`), both with 1-dim edges, and
`source_optuna_hps/` holds the per-task tuned hyperparameters (the search was unseeded, so
these values cannot be recovered by re-running it).

The v1 label CSVs keep stale absolute paths, which is harmless: both consumers glob the `.pt`
files directly rather than reading `graph_path`. The split tables key on `slide_name`.

### Batch effects: does the encoder embed biology or hospital?

LISI on TCGA-BRCA slide embeddings (PCA-50, perplexity 30) plus a 1,000-permutation test.
All scaled metrics are **higher = better**.

| Model | scaled iLISI | scaled cLISI | silhouette subtype | observed iLISI | permutation null | p |
|---|:--:|:--:|:--:|:--:|:--:|:--:|
| DINOv2 | 0.0894 | 0.6579 | −0.0263 | 4.3957 | 8.3035 | < 0.001 |
| MAE | 0.0306 | **0.7107** | −0.0119 | 2.1623 | 7.5683 | < 0.001 |
| **GrapHist v1** | **0.0923** | 0.7101 | **+0.0105** | **4.5084** | 7.4747 | < 0.001 |

**All three carry a significant batch effect.** This is a ranking, not a clean bill of health.
GrapHist retains 60.3 % of the permutation null's site mixing against 52.9 % and 28.6 %, and is
the only model with a positive subtype silhouette, i.e. the only one separating IDC from ILC at
all. `p < 0.001` rather than 0: the estimator is `mean(null ≤ observed)` over 1,000
permutations, so 0 is a resolution floor.

This study has no data directory of its own: it runs on `baselines/` and
`labels/tcga_brca_slide_labels.csv`, both of which ship here, and the analysis code is in the
code repository.

### `studies/preprocessing_runtime/`: cost of building the graphs

19,200 tiles through all three preprocessing stages, shipped as one tar per stage
(38,400 segmentation outputs, 19,009 feature CSVs, 14,341 graphs) plus the full run log.

| Stage | Device | s/patch |
|---|---|--:|
| Cell segmentation | CPU | 0.0594 |
| Cell feature extraction | GPU | 0.1550 |
| Graph construction | CPU | 0.0003 |
| **Total** | | **0.2147** |

Read as "this pipeline on this machine", not as a benchmark: the device assignment is unusual
(segmentation ran CPU-only), the hardware is a consumer desktop, and the wrapper that produced
the wall-clock numbers is not included, so the timing boundary cannot be audited. The stage
tars do let you check how many tiles survived each step: 19,200 in, 14,341 graphs out (74.7 %),
the losses being tiles with no detections and then patches with fewer than 10 cells.

---

## upstream_v1/

Byte-identical mirror of the original GrapHist v1 dataset releases, so this repository does not
depend on another account staying available: `bracs.tar` (4,493 graphs), `breakhis.tar` (522),
and `tcga_brca/` (254 files, 271 GB). These carry **1-dimensional** `edge_attr` and will not
load into `graphist_v2.pt`. The README inside `tcga_brca/` is the original author's and is left
untouched.

---

## Reproducing the published numbers

The training, evaluation and analysis scripts referenced below live in
[github.com/Ace3Z/GrapHist-V2](https://github.com/Ace3Z/GrapHist-V2). This repository ships
the data they consume, plus the model and its loader.

Run the `rebase_graph_paths.py` step first in every case.

**Slide-level MIL subtyping**

```bash
python src/train/generate_embs.py \
  --dataset BACH --scale slide --seed 0 \
  --checkpoint_path graphist_v2.pt \
  --sample_data_folder  graphs/bach/extracted \
  --sample_split_folder graphs/bach \
  --scale_vals_path     graphs/bach/normalization.json \
  --output_dir          out/bach \
  --encoder acm_gineconv --decoder acm_gineconv \
  --num_hidden 512 --num_layers 5 --concat_hidden True \
  --encoder_norm layer --input_norm none \
  --normalize_input True --input_min_std 0.01 --input_clip 10 \
  --edge_distance_in_proj False \
  --mask_rate 0.5 --replace_rate 0.1 --alpha_l 3 --activation prelu
```

Then run the MIL evaluation on the resulting embeddings. `--edge_distance_in_proj False`,
`--encoder_norm layer` and `--concat_hidden True` are load-critical, as the model card explains.
Or skip inference and use `embeddings/slide/` directly.

**Survival analysis (TCGA-BRCA)**

```bash
python studies/survival/prepare_dataframe.py \
  --embeddings_dir embeddings/slide/TCGA_BRCA \
  --clinical_tsv   labels/tcga_brca_clinical.tsv \
  --output_csv     survival.csv
python studies/survival/investigate.py \
  --input_csv survival.csv --output_dir out/survival --model_type graphist
```

Swap `--embeddings_dir` for any directory under `baselines/` to get the comparison rows.
Requires `lifelines`.

**SPIDER-breast patch-level probe.** Same as slide-level but with `--scale patch` and
`--sample_data_folder graphs/spider_breast` (which holds the CSVs; the graph shards are in
`graphs/spider_breast/data/`), then a
logistic-regression probe on the frozen embeddings using `metadata.csv`.

**Cell-type identification.** Use `embeddings/cell/` directly with a
StandardScaler + logistic-regression probe as described above.

### What cannot be reproduced from this repository

- **Raw images.** Only derived cell-level features are released; the source cohorts must be
  obtained from their providers.
- **The pseudo-label validation gate** needs the external CellViT++ repository and SAM-H
  weights.
- **Hardware timings** are wall-clock measurements on specific GPUs.

## Datasets used

This release is built from seven public cohorts. If you use it, please cite GrapHist **and**
the source cohort(s) your work touches.

| Cohort | Used for | Source |
|---|---|---|
| **TCGA-BRCA** | pre-training (11.1 M graphs), slide-level evaluation, survival | [GDC Data Portal](https://portal.gdc.cancer.gov/) |
| **BACH** (ICIAR 2018) | slide-level subtyping, homophily | [Grand Challenge](https://iciar2018-challenge.grand-challenge.org/) |
| **BRACS** | slide-level subtyping, homophily | [bracs.icar.cnr.it](https://www.bracs.icar.cnr.it/) |
| **BreakHis** | slide-level subtyping, homophily | [P&D Lab, UFPR](https://web.inf.ufpr.br/vri/databases/breast-cancer-histopathological-database-breakhis/) |
| **NuCLS** | cell-type identification, homophily | [NuCLS](https://sites.google.com/view/nucls/home) |
| **PanNuke** | cell-type identification, homophily | [TIA Centre, Warwick](https://warwick.ac.uk/fac/sci/dcs/research/tia/data/pannuke/) |
| **SPIDER-breast** | patch-level subtyping | [histai/SPIDER-breast](https://huggingface.co/datasets/histai/SPIDER-breast) |

<details>
<summary>BibTeX for the source cohorts</summary>

```bibtex
@article{weinstein2013cancer,
  title={The cancer genome atlas pan-cancer analysis project},
  author={Weinstein, John N and Collisson, Eric A and Mills, Gordon B and Shaw, Kenna R and
          Ozenberger, Brad A and Ellrott, Kyle and Shmulevich, Ilya and Sander, Chris and
          Stuart, Joshua M},
  journal={Nature Genetics}, volume={45}, number={10}, pages={1113--1120}, year={2013},
  publisher={Nature Publishing Group}
}

@article{aresta2019bach,
  title={{BACH}: Grand challenge on breast cancer histology images},
  author={Aresta, Guilherme and Ara{\'u}jo, Teresa and Kwok, Scotty and
          Chennamsetty, Sai Saketh and Safwan, Mohammed and Alex, Varghese and others},
  journal={Medical Image Analysis}, volume={56}, pages={122--139}, year={2019},
  publisher={Elsevier}
}

@article{brancati2022bracs,
  title={{BRACS}: A Dataset for BReAst Carcinoma Subtyping in {H\&E} Histology Images},
  author={Brancati, Nadia and Anniciello, Anna Maria and Pati, Pushpak and Riccio, Daniel and
          Scognamiglio, Giosu{\`e} and Jaume, Guillaume and De Pietro, Giuseppe and
          Di Bonito, Maurizio and Foncubierta, Antonio and Botti, Gerardo and others},
  journal={Database}, volume={2022}, pages={baac093}, year={2022},
  publisher={Oxford University Press UK}
}

@article{spanhol2015dataset,
  title={A dataset for breast cancer histopathological image classification},
  author={Spanhol, Fabio A and Oliveira, Luiz S and Petitjean, Caroline and Heutte, Laurent},
  journal={IEEE Transactions on Biomedical Engineering}, volume={63}, number={7},
  pages={1455--1462}, year={2015}, publisher={IEEE}
}

@article{amgad2022nucls,
  title={{NuCLS}: A scalable crowdsourcing approach and dataset for nucleus classification and
         segmentation in breast cancer},
  author={Amgad, Mohamed and Atteya, Lamees A and Hussein, Hagar and Mohammed, Kareem Hosny and
          Hafiz, Ehab and Elsebaie, Maha AT and Alhusseiny, Ahmed M and
          AlMoslemany, Mohamed Atef and Elmatboly, Abdelmagid M and Pappalardo, Philip A and others},
  journal={GigaScience}, volume={11}, pages={giac037}, year={2022},
  publisher={Oxford University Press}
}

@article{gamper2020pannuke,
  title={{PanNuke} dataset extension, insights and baselines},
  author={Gamper, Jevgenij and Koohbanani, Navid Alemi and Benes, Ksenija and Graham, Simon and
          Jahanifar, Mostafa and Khurram, Syed Ali and Azam, Ayesha and Hewitt, Katherine and
          Rajpoot, Nasir},
  journal={arXiv preprint arXiv:2003.10778}, year={2020}
}

@article{nechaev2025spider,
  title={{SPIDER}: A Comprehensive Multi-Organ Supervised Pathology Dataset and Baseline Models},
  author={Nechaev, Dmitry and Pchelnikov, Alexey and Ivanova, Ekaterina},
  year={2025}, eprint={2503.02876}, archivePrefix={arXiv}, primaryClass={cs.CV}
}
```
</details>

## Licence

Released `cc-by-nc-sa-4.0`. SPIDER-breast is `cc-by-nc-4.0`, research use only, and those
terms travel with the derived graphs:

`labels/tcga_brca_clinical.tsv` is the open-access GDC clinical export, redistributed under
TCGA's open-access terms. The source cohorts keep their own licences, so cite their papers
alongside GrapHist.

This work was done by [**Mahbod Tajdini**](https://mahbodtajdini.com) and [**Tomás Gadea Alcaide**](https://tomasgadea.com/), supervised by members of
[LTS4, EPFL](https://www.epfl.ch/labs/lts4/).