| # CLAUDE.md |
|
|
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
|
|
| ## Project Overview |
|
|
| Medical imaging data engineering pipeline for standardizing diverse datasets (CT, MRI, PET) into a unified NIfTI format with consistent JSON metadata. Each subdirectory handles one dataset (AbdomenAtlas, BRATS, MnM2, OASIS, OAI_ZIB, PSMA, Kaggle OSIC, etc.). |
| |
| ## Running Data Cleaning Scripts |
| |
| Each dataset has its own `dataclean_*.py` script. Run from the dataset's subdirectory: |
| |
| ```bash |
| python dataclean_abdomen_atlas.py --target_path /path/to/raw/data --output_dir /path/to/output |
| ``` |
| |
| All scripts follow the same `--target_path` / `--output_dir` argument pattern. Versioned scripts (e.g., `_v2.py`, `_v3.py`) represent iterative improvements; use the highest version unless investigating regressions. |
| |
| ## Dependencies |
| |
| Python 3 with: `SimpleITK`, `pandas`, `numpy`, `tqdm`, `openpyxl` (for Excel metadata). No requirements.txt exists — install manually. |
| |
| ## Architecture |
| |
| ### Processing Pipeline (per dataset) |
| |
| 1. **Load** raw data (DICOM via `sitk.ImageSeriesReader`, NIfTI via `sitk.ReadImage`, or NRRD) |
| 2. **Extract metadata** from headers, CSV files, or DICOM tags |
| 3. **Resample** to isotropic spacing using minimum voxel spacing (`get_unisize_resampler`) |
| 4. **Clamp intensities** — CT: `[-300, 300]` HU; MRI: varies per dataset |
| 5. **Process segmentation labels** with identical resampling (nearest-neighbor interpolation) |
| 6. **Validate** image/label dimension alignment via `assert` on `GetSize()` |
| 7. **Write** standardized NIfTI (`.nii.gz`) + append to `nifti_mappings.json` |
| |
| ### Key Shared Components |
| |
| **`util.py`** (copied into each dataset directory — not a shared import): |
| - `meta_data` class — validates metadata against `config_format.json` schema, enforces required fields (Modality, OriImg_path, Spacing_mm, Size, Dataset_name), normalizes ambiguous terminology via synonym dictionaries |
| - `get_unisize_resampler()` — builds a SimpleITK resampler for isotropic spacing; returns `None` if spacing is already isotropic |
| - `clamp_image()` — HU/intensity clamping via `sitk.ClampImageFilter` |
| - `get_synonyms_dict()` / `replace_synonyms()` — canonical mapping for ROI names, tissue labels, modalities, and task types |
| - `load_nifti()`, `load_dicom_images()`, `save_nifti()` — I/O wrappers that embed `FolderPath` metadata in NIfTI headers |
| |
| **`config_format.json`** (per dataset directory): defines the metadata schema — field types, required flags, and allowed option values. |
| |
| ### Output Structure |
| |
| ``` |
| {output_dir}/{patient_id}/{patient_id}.nii.gz # processed image |
| {output_dir}/{patient_id}/{task}/{tissue}.nii.gz # segmentation labels |
| {output_dir}/nifti_mappings.json # metadata keyed by output path |
| {output_dir}/failed_files.json # files that failed processing |
| ``` |
| |
| ### Dataset-Specific Notes |
| |
| - **AbdomenAtlas**: 25-organ segmentation labels stored as individual NIfTI files per organ; also has `combined_labels.nii.gz` (values 0-25) |
| - **BRATS (2019/2020/2021)**: Multi-modal MRI (FLAIR, T1, T1ce, T2) — each modality processed as a separate sub-modality entry |
| - **MnM2/MnMs**: Cardiac MRI with vendor metadata (Siemens, Philips, GE, Canon) |
| - **OASIS**: Both cross-sectional and longitudinal variants; includes clinical scores (MMSE, CDR) |
| - **OAI_ZIB**: Knee MRI with 6-structure segmentation and clinical grading (WOMAC) |
| - **PSMA**: Dual-tracer PET/CT (PSMA & FDG); has longitudinal variant |
| |
| ## Important Conventions |
| |
| - Resampling uses the **minimum** of the original spacing values to create isotropic voxels |
| - Labels are resampled with **nearest-neighbor** interpolation; images use **linear** |
| - The `meta_data` class normalizes terminology automatically — e.g., "chest" maps to "thorax", "seg" maps to "segmentation" |
| - `util.py` is duplicated across directories (not shared via import) — changes must be propagated manually |
| - Code comments and docstrings are frequently in Chinese |
| - Log files (`*.log`) in each directory contain processing run history — these can be large (up to 23 MB) |
|
|
