docs/data_inventory.md

# Data Inventory

## Tahoe-100M

### Core data (Tier 1 — ~42 GB)
| File | Path | Size | What |
|---|---|---|---|
| pdex | `/nvme-shared/Data/100m_h5ad/pdex_results/all_plates_pdex.parquet` | 40.6 GB | FC, p-value, FDR, ref_mean (long format) |
| cell_eval 2K | `/nvme-shared/Data/tahoe_100m_de_cell_eval/plate_plate*.parquet` (14 files) | 1.0 GB | Expression deltas, 2000 HVG genes |

### Full gene set (Tier 2 — ~15 GB additional)
| File | Path | Size |
|---|---|---|
| cell_eval full | `/nvme-shared/Data/tahoe_100m_de_cell_eval_full/plate*_full_filtered.parquet` (14 files) | 15.3 GB |
| Per-cell-line gene list | `/nvme-shared/drive_3/ml/pod_diffusion/Data/top3k_genes_per_cellline_remapped_v2.json` | ~3 MB |

### A/B variants (Tier 3 — ~88 GB additional)
| File | Path | Size |
|---|---|---|
| pdex group A | `/nvme-shared/Data/100m_h5ad_A/pdex_results/all_plates_pdex.parquet` | 43.4 GB |
| pdex group B | `/nvme-shared/Data/100m_h5ad_B/pdex_results/all_plates_pdex.parquet` | 43.5 GB |
| cell_eval group A | `/nvme-shared/drive_3/ANALYSIS/analysis_255/Scripts/sc_aggregate/results_tahoe100m_celleval/groupA/group_A_celleval_plate*.parquet` (14 files) | 0.7 GB |
| cell_eval group B | `/nvme-shared/drive_3/ANALYSIS/analysis_255/Scripts/sc_aggregate/results_tahoe100m_celleval/groupB/group_B_celleval_plate*.parquet` (14 files) | 0.7 GB |

### Sensitivity labels
| File | Path | Size | Coverage |
|---|---|---|---|
| seqrun growth_rate | `/nvme-shared/Data/seqrun_26_27_merged/5/qc/generic/growth_rate_long.parquet` | small | 282 (cell, treatment) pairs |
| PRISM dose-matched | `/nvme-shared/Data/prism/dose_matched_pdex_prism.parquet` | small | 9,240 (cell, drug, dose) rows, 35 cells x 88 drugs x 3 doses |

### Raw h5ad files
| File | Path | Size |
|---|---|---|
| Plate h5ads | `/nvme-shared/Data/100m_h5ad/plate{1..14}_full_filtered.h5ad.gz` | ~40 GB each |

## Parse PBMC

### Core data
| File | Path | Size |
|---|---|---|
| pdex | `/nvme-shared/Data/parse_pbmc/by_cell_type/pdex_results/all_celltypes_pdex.parquet` | 11.6 GB |
| cell_eval | `/nvme-shared/Data/parse_pbmc/by_cell_type/delta_results/all_celltypes_delta.parquet` | 2.1 GB |

### A/B variants
| File | Path | Size |
|---|---|---|
| pdex A | `/nvme-shared/Data/parse_pbmc/by_cell_type_A/pdex_results/all_celltypes_pdex.parquet` | 10.1 GB |
| cell_eval A | `/nvme-shared/Data/parse_pbmc/by_cell_type_A/delta_results/all_celltypes_delta.parquet` | 1.8 GB |
| pdex B | `/nvme-shared/Data/parse_pbmc/by_cell_type_B/pdex_results/all_celltypes_pdex.parquet` | 10.2 GB |
| cell_eval B | `/nvme-shared/Data/parse_pbmc/by_cell_type_B/delta_results/all_celltypes_delta.parquet` | 1.8 GB |

## PerturbAI

### Core data
| File | Path | Size |
|---|---|---|
| pdex (pooled) | `/nvme-shared/Data/perturbai/wholebrain_crispr_atlas/h5ads/pdex_results_pooled/all_pdex.parquet` | 23.7 GB |
| cell_eval (pooled) | `/nvme-shared/Data/perturbai/wholebrain_crispr_atlas/h5ads/delta_results_pooled/all_delta.parquet` | 4.0 GB |

## Upstream scripts (generate parquets from h5ad)

| Script | Location | What |
|---|---|---|
| Tahoe pdex | `/nvme-shared/Data/100m_h5ad/run_pdex.py` (original) / `scripts/data_prep/compute_pdex.py` (in-repo) | Mann-Whitney via pdex library |
| Tahoe cell_eval | `Barotaxis2/vector_diffusion/scripts/data_prep/compute_celleval_lfc.py` (original) / `scripts/data_prep/compute_celleval_deltas.py` (in-repo) | Pseudobulk mean - DMSO |
| Parse pdex | `/nvme-shared/Data/parse_pbmc/by_cell_type/run_pdex.py` | Per-donor parallelized |
| Parse cell_eval | `/nvme-shared/Data/parse_pbmc/by_cell_type/compute_delta.py` | normalize_total(3320) + log1p |

## Gene lists

| List | Path | Count | Notes |
|---|---|---|---|
| Tahoe static 2K | `splits/tahoe/static_2k_genes.json` | 2,000 | Current model gene set |
| Parse static 2K | `splits/parse/static_2k_genes.json` | 2,000 | 530 overlap with Tahoe |
| PerturbAI static 2K | `splits/perturbai/static_2k_genes.json` | 2,000 | Mouse genes |
| Per-cell-line top 3K | `/nvme-shared/drive_3/ml/pod_diffusion/Data/top3k_genes_per_cellline_remapped_v2.json` | 28,661 union | 1,261-18,956 per cell |
| pdex total | (in parquet) | 62,710 | All available |
| cell_eval full | (in parquet columns) | 62,710 | Matches pdex |

## S3 upload tiers (from issue #3)

| Tier | Cumulative | What you can do |
|---|---|---|
| Tier 1 | ~42 GB | Reproduce headline model + metrics |
| +Tier 2 | ~57 GB | Broader gene lists (up to 62K) |
| +Tier 3 | ~145 GB | Noise-ceiling A/B calibration |
| +Tier 4 | ~370 GB | Cross-dataset (Parse, PerturbAI) |