DATA.md

Datasets, embeddings and studies

Everything in this repository, what it is, and what it is for. Counts were read back from the Hugging Face API after upload.

Directory	Contents
graphs/	cell graphs, the model's inputs
embeddings/	embeddings produced by graphist_v2.pt
baselines/	DINOv2, MAE and GrapHist v1 features for comparison
labels/	TCGA-BRCA clinical table and slide labels
studies/	side analyses, each self-contained
upstream_v1/	mirror of the original GrapHist v1 releases

---

⚠️ Read this before using graphs/

The label CSVs store bare filenames in their graph_path column, and the loader passes that column to torch.load unchanged, so it resolves against the process working directory rather than against the CSV. So unless you happen to run from inside the graph folder, every path fails, and the failure is quiet: unreadable paths are dropped with a single WARNING: Filtered out <n> missing/empty graph files, leaving you with a smaller dataset, or an empty one, which surfaces later as an unhelpful IndexError.

Fix it once per dataset, after extracting, with the helper in modeling/:

python modeling/rebase_graph_paths.py --csv <path to sample_labels_rich.csv> \
       --mode absolute --root "$PWD/<path to the .pt files>"

It rewrites by basename, so it does not matter where the graphs ended up, and it is idempotent. Verify before training:

python modeling/rebase_graph_paths.py --csv <csv> --check     # expect N/N

Expected: BACH 14,341, BreakHis 709, BRACS 96,153, SPIDER-breast 71,745, TCGA-BRCA 11,149,499. Anything less means graphs are being silently skipped.

Graph format

Tensor	Shape	Meaning
x	[num_nodes, 96]	per-cell morphology, texture and colour features
edge_index	[2, num_edges]	Delaunay edges, pruned at 100 µm
edge_attr	[num_edges, 75]	column 0 = centroid distance (µm); columns 1–74 describe the inter-cellular region

Every dataset ships the same four companion files, and all four are needed. .pt graphs alone are not enough:

File	Purpose
sample_labels_rich.csv	graph → sample id → label (the file you rebase)
sample_split.csv	train / test assignment
sample_labels.csv	slide-level labels
normalization.json	per-feature mean/std for NormalizeData

---

graphs/

Large datasets ship as tars because Hugging Face documents a ceiling of 10,000 entries per folder.

graphs/tcga_brca/: 11,149,499 graphs, 652 GB

The pre-training corpus. 324 × ~2.0 GB tars under tcga_brca/data/, plus the four companion files (its sample_labels_rich.csv is 1.94 GB). Also the in-domain cohort for slide-level evaluation.

Run from the repository root:

mkdir -p graphs/tcga_brca/extracted
for t in graphs/tcga_brca/data/tcga-brca-*.tar; do tar -xf "$t" -C graphs/tcga_brca/extracted; done
python modeling/rebase_graph_paths.py --csv graphs/tcga_brca/sample_labels_rich.csv \
       --mode absolute --root "$PWD/graphs/tcga_brca/extracted"

graphs/bach/: 14,341 graphs

BACH (ICIAR 2018), 224 px @ 20×, 4-class. One graphs.tar (flat: the .pt files and sample_labels_rich.csv) plus the three other companion files. Extract into a directory you create yourself:

mkdir -p graphs/bach/extracted && tar -xf graphs/bach/graphs.tar -C graphs/bach/extracted
python modeling/rebase_graph_paths.py --csv graphs/bach/extracted/sample_labels_rich.csv \
       --mode absolute --root "$PWD/graphs/bach/extracted"

graphs/bracs/: 96,153 graphs

BRACS, 7-class lesion subtyping. 6 × ~1 GB tars under bracs/data/ holding only .pt files. Unlike BACH, the sample_labels_rich.csv sits one level up at bracs/.

graphs/breakhis/: 709 graphs

BreakHis, binary benign/malignant. Small enough to ship as loose .pt files under breakhis/graphs/, with sample_labels_rich.csv alongside them.

graphs/spider_breast/: 71,745 graphs

SPIDER-breast, 18 tumour-subtype classes, train 62,995 / test 8,750. 18 × ~245 MB tars under spider_breast/data/ (.pt only; the CSV is one level up).

Two things to know:

• The label files cover 92,892 patches, the graphs are 71,745. Graph construction skips any patch with fewer than 10 detected cells, which removes cell-sparse tissue almost entirely: Fat retains 7 of 6,286 patches (1 in test), Fibrosis 415 of 6,260, Necrosis 2,147 of 5,396, while ten other classes lose ~0 %. In an 18-class macro-F1 this matters, because Fat rests on a single test patch. metadata.csv is the file that matches the graphs exactly.
• sample_id is doubled in metadata.csv (patch_0000017_patch_0000017, matching the .pt filename) but single elsewhere. A naive join between them matches 0 rows; strip the doubling first. Do not "normalise" metadata.csv: the doubled form is what the embedding filenames need.

---

embeddings/

Produced by graphist_v2.pt. Use these to skip inference entirely.

embeddings/slide/: 6,531 slides

Per-slide embeddings.h5, dataset key embeddings, shape (n_tiles, 512): BACH 397, BRACS 4,493, BreakHis 522, TCGA-BRCA 1,119.

Feeds slide-level MIL subtyping and the TCGA-BRCA survival analysis (tiles → slide → patient mean-pool, then Cox proportional hazards).

Each cohort also carries one train/normalizer_values.json. Keep it. The loader resolves <data>/../train/normalizer_values.json, and if it is missing it silently recomputes the statistics from whatever split you point at and writes the file, which changes your numbers and requires a writable directory.

embeddings/cell/: 22 tars

Per-graph .npz (keys embedding (n, 512), label (n,)) for cell-type identification: NuCLS_{main,super}_fold1-5, PanNuke_{20x,40x}_test1-3, PanNuke_breast_{20x,40x}_test1-3. Evaluated with StandardScaler + multinomial logistic regression (class_weight=balanced, max_iter=2000, seed 0).

These are the only route to the cell-level results: 75-dim NuCLS/PanNuke graphs were never built, and the 1-dim v1 graphs are not a substitute (the encoder slices edge_attr[:, 1:] into a Linear(74, 512), so a 1-dim edge attribute is a shape error).

One known gap: every NuCLS tar holds 1,693 .npz against 1,694 graphs, and the same single graph is absent from all ten. It is TCGA-S3-AA15-DX1_id-5ea40a6addda5f839898f24a_left-57268_top-29680_bottom-29958_right-57547. Effect on macro-F1 ≈ 0.06 %.

---

baselines/

Tile features from other encoders on TCGA-BRCA, for the comparison rows in survival analysis and batch-effect probing. Same .h5 layout as embeddings/slide/.

Directory	Model	Slides
baselines/dinov2/	DINOv2 ViT-S/14	1,126
baselines/mae/	MAE ViT-S	1,126
baselines/graphist_v1/embeddings/	GrapHist v1	1,122

baselines/graphist_v1/graphist_v1.pt is the vanilla GrapHist v1 checkpoint (ACM-GIN, 1-dim edges, 114,203,790 B). It is the fine-tuning starting point for the AdapterGNN study and the encoder behind the v1 comparison numbers.

Slide counts differ slightly (1,126 / 1,122 / 1,119 for v2) because the embedding sets cover marginally different slides. The survival scripts join on patient id, so each comparison runs over its own intersection.

---

labels/

File	Contents
tcga_brca_clinical.tsv	GDC open-access clinical export, 5,546 rows over 1,098 patients
tcga_brca_slide_labels.csv	slide → label, header exactly sample_id,label

The clinical table supplies the survival time / event columns; without it no survival number can be computed. Four columns are read: cases.submitter_id, demographic.vital_status, demographic.days_to_death, diagnoses.days_to_last_follow_up.

There is no site/hospital column. The batch-effect analysis derives the hospital from field 1 of the TCGA barcode.

Note on joins: for BACH, BreakHis and BRACS the label file is a superset of the split file (every split id has a label). TCGA-BRCA is the exception: 1,122 labels against 1,126 split rows, so four split ids have no label row and an inner join silently drops them.

---

studies/

Side analyses. Each has its own folder here holding the data, the code that produced it and its outputs, except the batch-effect study, which reuses baselines/ and labels/ and so ships no folder of its own.

studies/homophily/: how homophilic are cell graphs?

H_node is the fraction of a cell's neighbours sharing its label. The fat tail is the share of cells below a threshold, i.e. cells whose neighbourhood is label-mixed, which a homophily-assuming GNN will smooth incorrectly.

Per-node tails over the pooled graph, isolated nodes excluded:

Cohort	labels	nodes	H_node < 0.3	< 0.5	pooled H_edge
NuCLS-super	ground truth, 4-class	52,017	5.55 %	10.77 %	0.8287
NuCLS-main	ground truth, 7-class	52,017	7.93 %	14.32 %	0.7931
PanNuke 20×	ground truth, 5-class	162,241	10.46 %	16.83 %	0.7590
PanNuke 40×	ground truth, 5-class	162,241	10.48 %	16.83 %	0.7589
BACH	CellViT++ pseudo, 4-class	439,281	17.79 %	28.62 %	0.6304
BRACS	CellViT++ pseudo, 4-class	4,301,987	14.56 %	23.31 %	0.6917
BreakHis	CellViT++ pseudo, 4-class	6,255	3.60 %	4.73 %	0.9299

The two views disagree, and that is the finding: pooled edge homophily is 0.76–0.83 on the labelled cohorts, so the graphs are globally homophilic, yet 5.6–10.5 % of individual cells sit below H_node 0.3. The mean hides a heterophilic minority. Label granularity drives part of it (NuCLS 4-class 5.55 % → 7-class 7.93 % on the same graphs). The per-graph tail is below 1 % everywhere, so this is not a handful of pathological graphs.

H_edge above is pooled (micro-averaged over the disjoint union), not the per-graph macro-mean, which runs 0.75–0.82 on the labelled cohorts.

Contents: tiles/ holds 109,904 224 px tile graphs with CellViT++ pseudo-labels (BACH 13,837, BRACS 95,670, BreakHis 397); bach_cellvit/ holds 400 per-image pseudo-label CSVs (538,421 cells, columns centroid_x_px,centroid_y_px,cell_type,type_prob) plus a summary JSON; and paper_metrics/ holds the pooled edge-homophily and pseudo-label-gate results as JSON.

The pseudo-labeller was validated on a held-out ground-truth split first: ARI 0.560 (threshold 0.5), Cohen κ 0.709, per-class confusion diagonal 0.864 / 0.562 / 0.900 / 0.000. Three of four classes pass, and the fourth (other_nucleus) is untestable at 7 ground-truth instances. Re-running that gate needs the external CellViT++ repository and SAM-H weights, which are not included here.

One caveat if you re-run the analysis: _strip_virtual_node removes the synthetic virtual node by degree (deg >= 0.95*(n-1)), which on small graphs also deletes real cells, up to 676 nodes in a cohort, without warning. It shifts the tails by at most 0.037 pp, so no conclusion changes, but it needs a minimum-degree guard.

studies/adaptergnn/: parameter-efficient fine-tuning

Adapters inserted into a frozen GrapHist v1 encoder for cell-type identification. Test macro-F1 %, each the mean over cross-validation folds (2 to 5 folds depending on the column):

Model	P20 Breast	P20 PanCancer	P40 Breast	P40 PanCancer	NuCLS main	NuCLS super
Supervised graph, ACM-bio	56.61	62.73	57.06	65.80	22.13	37.35
Supervised graph, ACM-UNI	58.05	67.63	56.81	66.99	21.68	39.46
DINOv2 probe	54.82	50.49	53.86	49.27	21.42	41.17
MAE probe	47.71	54.88	47.54	55.26	25.19	45.31
GrapHist v1 frozen probe	55.26	58.78	56.43	59.47	26.57	46.24
GrapHist v1 + AdapterGNN (shared recipe)	57.54	65.86	57.07	64.92	27.22	44.47
GrapHist v1 + AdapterGNN (tuned)	58.42	66.79	59.19	70.93	27.55	48.65

The tuned protocol takes 5 of 6 columns. Against the frozen probe, which uses the same encoder, so the comparison isolates the adapters, it gains +3.2 / +8.0 / +2.8 / +11.5 / +1.0 / +2.4.

Contents: graphs_v1/ holds NuCLS (1,694 .pt × 2 label granularities plus 5-fold split tables) and PanNuke (3 folds × 20×/40×, 14,414 .pt), both with 1-dim edges, and source_optuna_hps/ holds the per-task tuned hyperparameters (the search was unseeded, so these values cannot be recovered by re-running it).

The v1 label CSVs keep stale absolute paths, which is harmless: both consumers glob the .pt files directly rather than reading graph_path. The split tables key on slide_name.

Batch effects: does the encoder embed biology or hospital?

LISI on TCGA-BRCA slide embeddings (PCA-50, perplexity 30) plus a 1,000-permutation test. All scaled metrics are higher = better.

Model	scaled iLISI	scaled cLISI	silhouette subtype	observed iLISI	permutation null	p
DINOv2	0.0894	0.6579	−0.0263	4.3957	8.3035	< 0.001
MAE	0.0306	0.7107	−0.0119	2.1623	7.5683	< 0.001
GrapHist v1	0.0923	0.7101	+0.0105	4.5084	7.4747	< 0.001

All three carry a significant batch effect. This is a ranking, not a clean bill of health. GrapHist retains 60.3 % of the permutation null's site mixing against 52.9 % and 28.6 %, and is the only model with a positive subtype silhouette, i.e. the only one separating IDC from ILC at all. p < 0.001 rather than 0: the estimator is mean(null ≤ observed) over 1,000 permutations, so 0 is a resolution floor.

This study has no data directory of its own: it runs on baselines/ and labels/tcga_brca_slide_labels.csv, both of which ship here, and the analysis code is in the code repository.

studies/preprocessing_runtime/: cost of building the graphs

19,200 tiles through all three preprocessing stages, shipped as one tar per stage (38,400 segmentation outputs, 19,009 feature CSVs, 14,341 graphs) plus the full run log.

Stage	Device	s/patch
Cell segmentation	CPU	0.0594
Cell feature extraction	GPU	0.1550
Graph construction	CPU	0.0003
Total		0.2147

Read as "this pipeline on this machine", not as a benchmark: the device assignment is unusual (segmentation ran CPU-only), the hardware is a consumer desktop, and the wrapper that produced the wall-clock numbers is not included, so the timing boundary cannot be audited. The stage tars do let you check how many tiles survived each step: 19,200 in, 14,341 graphs out (74.7 %), the losses being tiles with no detections and then patches with fewer than 10 cells.

---

upstream_v1/

Byte-identical mirror of the original GrapHist v1 dataset releases, so this repository does not depend on another account staying available: bracs.tar (4,493 graphs), breakhis.tar (522), and tcga_brca/ (254 files, 271 GB). These carry 1-dimensional edge_attr and will not load into graphist_v2.pt. The README inside tcga_brca/ is the original author's and is left untouched.

---

Reproducing the published numbers

The training, evaluation and analysis scripts referenced below live in github.com/Ace3Z/GrapHist-V2. This repository ships the data they consume, plus the model and its loader.

Run the rebase_graph_paths.py step first in every case.

Slide-level MIL subtyping

python src/train/generate_embs.py \
  --dataset BACH --scale slide --seed 0 \
  --checkpoint_path graphist_v2.pt \
  --sample_data_folder  graphs/bach/extracted \
  --sample_split_folder graphs/bach \
  --scale_vals_path     graphs/bach/normalization.json \
  --output_dir          out/bach \
  --encoder acm_gineconv --decoder acm_gineconv \
  --num_hidden 512 --num_layers 5 --concat_hidden True \
  --encoder_norm layer --input_norm none \
  --normalize_input True --input_min_std 0.01 --input_clip 10 \
  --edge_distance_in_proj False \
  --mask_rate 0.5 --replace_rate 0.1 --alpha_l 3 --activation prelu

Then run the MIL evaluation on the resulting embeddings. --edge_distance_in_proj False, --encoder_norm layer and --concat_hidden True are load-critical, as the model card explains. Or skip inference and use embeddings/slide/ directly.

Survival analysis (TCGA-BRCA)

python studies/survival/prepare_dataframe.py \
  --embeddings_dir embeddings/slide/TCGA_BRCA \
  --clinical_tsv   labels/tcga_brca_clinical.tsv \
  --output_csv     survival.csv
python studies/survival/investigate.py \
  --input_csv survival.csv --output_dir out/survival --model_type graphist

Swap --embeddings_dir for any directory under baselines/ to get the comparison rows. Requires lifelines.

SPIDER-breast patch-level probe. Same as slide-level but with --scale patch and --sample_data_folder spider_breast (the folder holding the CSV, not the graphs), then a logistic-regression probe on the frozen embeddings using metadata.csv.

Cell-type identification. Use embeddings/cell/ directly with a StandardScaler + logistic-regression probe as described above.

What cannot be reproduced from this repository

• Raw images. Only derived cell-level features are released; the source cohorts must be obtained from their providers.
• The pseudo-label validation gate needs the external CellViT++ repository and SAM-H weights.
• Hardware timings are wall-clock measurements on specific GPUs.

Datasets used

This release is built from seven public cohorts. If you use it, please cite GrapHist and the source cohort(s) your work touches.

Cohort	Used for	Source
TCGA-BRCA	pre-training (11.1 M graphs), slide-level evaluation, survival	GDC Data Portal
BACH (ICIAR 2018)	slide-level subtyping, homophily	Grand Challenge
BRACS	slide-level subtyping, homophily	bracs.icar.cnr.it
BreakHis	slide-level subtyping, homophily	P&D Lab, UFPR
NuCLS	cell-type identification, homophily	NuCLS
PanNuke	cell-type identification, homophily	TIA Centre, Warwick
SPIDER-breast	patch-level subtyping	histai/SPIDER-breast

BibTeX for the source cohorts

@article{weinstein2013cancer,
  title={The cancer genome atlas pan-cancer analysis project},
  author={Weinstein, John N and Collisson, Eric A and Mills, Gordon B and Shaw, Kenna R and
          Ozenberger, Brad A and Ellrott, Kyle and Shmulevich, Ilya and Sander, Chris and
          Stuart, Joshua M},
  journal={Nature Genetics}, volume={45}, number={10}, pages={1113--1120}, year={2013},
  publisher={Nature Publishing Group}
}

@article{aresta2019bach,
  title={{BACH}: Grand challenge on breast cancer histology images},
  author={Aresta, Guilherme and Ara{\'u}jo, Teresa and Kwok, Scotty and
          Chennamsetty, Sai Saketh and Safwan, Mohammed and Alex, Varghese and others},
  journal={Medical Image Analysis}, volume={56}, pages={122--139}, year={2019},
  publisher={Elsevier}
}

@article{brancati2022bracs,
  title={{BRACS}: A Dataset for BReAst Carcinoma Subtyping in {H\&E} Histology Images},
  author={Brancati, Nadia and Anniciello, Anna Maria and Pati, Pushpak and Riccio, Daniel and
          Scognamiglio, Giosu{\`e} and Jaume, Guillaume and De Pietro, Giuseppe and
          Di Bonito, Maurizio and Foncubierta, Antonio and Botti, Gerardo and others},
  journal={Database}, volume={2022}, pages={baac093}, year={2022},
  publisher={Oxford University Press UK}
}

@article{spanhol2015dataset,
  title={A dataset for breast cancer histopathological image classification},
  author={Spanhol, Fabio A and Oliveira, Luiz S and Petitjean, Caroline and Heutte, Laurent},
  journal={IEEE Transactions on Biomedical Engineering}, volume={63}, number={7},
  pages={1455--1462}, year={2015}, publisher={IEEE}
}

@article{amgad2022nucls,
  title={{NuCLS}: A scalable crowdsourcing approach and dataset for nucleus classification and
         segmentation in breast cancer},
  author={Amgad, Mohamed and Atteya, Lamees A and Hussein, Hagar and Mohammed, Kareem Hosny and
          Hafiz, Ehab and Elsebaie, Maha AT and Alhusseiny, Ahmed M and
          AlMoslemany, Mohamed Atef and Elmatboly, Abdelmagid M and Pappalardo, Philip A and others},
  journal={GigaScience}, volume={11}, pages={giac037}, year={2022},
  publisher={Oxford University Press}
}

@article{gamper2020pannuke,
  title={{PanNuke} dataset extension, insights and baselines},
  author={Gamper, Jevgenij and Koohbanani, Navid Alemi and Benes, Ksenija and Graham, Simon and
          Jahanifar, Mostafa and Khurram, Syed Ali and Azam, Ayesha and Hewitt, Katherine and
          Rajpoot, Nasir},
  journal={arXiv preprint arXiv:2003.10778}, year={2020}
}

@article{nechaev2025spider,
  title={{SPIDER}: A Comprehensive Multi-Organ Supervised Pathology Dataset and Baseline Models},
  author={Nechaev, Dmitry and Pchelnikov, Alexey and Ivanova, Ekaterina},
  year={2025}, eprint={2503.02876}, archivePrefix={arXiv}, primaryClass={cs.CV}
}

Licence

Released cc-by-nc-sa-4.0. SPIDER-breast is cc-by-nc-4.0, research use only, and those terms travel with the derived graphs:

labels/tcga_brca_clinical.tsv is the open-access GDC clinical export, redistributed under TCGA's open-access terms. The source cohorts keep their own licences, so cite their papers alongside GrapHist.

This work was done by Mahbod Tajdini and Tomás Gadea Alcaide, supervised by members of LTS4, EPFL.