| --- |
| license: mit |
| tags: |
| - medical-imaging |
| - data-engineering |
| - preprocessing |
| - nifti |
| - dicom |
| - simpleitk |
| library_name: simpleitk |
| --- |
| |
| # Data_Engineering — Medical Imaging Cleanup Pipeline |
| |
| Standardize diverse medical imaging datasets (CT, MRI, PET) into a unified **NIfTI** format with consistent JSON metadata. Each subdirectory targets one dataset. |
| |
| > Companion repo to [`DRDMsig/Omini3D`](https://huggingface.co/DRDMsig/Omini3D) — produces the standardized data that OmniMorph trains on. |
| |
| ## Supported Datasets |
| |
| | Subdirectory | Dataset | Modality | |
| |---|---|---| |
| | `AbdomenAtlas/` | AbdomenAtlas | CT | |
| | `AbdomenCT1k/` | AbdomenCT-1K | CT | |
| | `brats2019_clean/` | BraTS 2019 | MRI (multi-sequence) | |
| | `brats2020_clean/` | BraTS 2020 | MRI (multi-sequence) | |
| | `brats2021_clean/` | BraTS 2021 | MRI (multi-sequence) | |
| | `kaggle_osic_clean/` | Kaggle OSIC Pulmonary Fibrosis | CT | |
| | `MnM2_clean/` | M&Ms-2 | Cardiac MRI | |
| | `MnMs_clean/` | M&Ms | Cardiac MRI | |
| | `OAISIS_clean/` | OASIS-1 / OASIS-2 | Brain MRI | |
| | `OAI_ZIB_clean/` | OAI-ZIB (knee) | MRI | |
| | `PSMA_clean/` | PSMA-FDG PET-CT (longitudinal) | PET + CT | |
| | `all/` | Cross-dataset utilities (artifact plane removal) | — | |
|
|
| Each cleaned dataset writes: |
|
|
| - Resampled & clamped `.nii.gz` images / segmentations |
| - Per-dataset `nifti_mappings.json` |
| - `failed_files.json` listing files the cleaner could not process |
|
|
| ## Repository Layout |
|
|
| ``` |
| <dataset>_clean/ |
| ├── dataclean_<dataset>.py # main cleanup script (use highest version: _v2.py, _v3.py, ...) |
| ├── util.py # shared helpers (copied per dir, not imported) |
| ├── config_format.json # metadata schema for `meta_data` validation |
| └── (optional) sample/, demo/ # tiny example NIfTI files for sanity checks |
| ``` |
|
|
| ## Usage |
|
|
| ```bash |
| cd AbdomenAtlas/ |
| python dataclean_abdomen_atlas_v2.py \ |
| --target_path /path/to/raw/AbdomenAtlas \ |
| --output_dir /path/to/output/AbdomenAtlas_clean |
| ``` |
|
|
| All scripts share the `--target_path` / `--output_dir` interface. Versioned scripts (`_v2.py`, `_v3.py`) supersede older versions; use the highest version unless investigating regressions. |
|
|
| ### Pipeline (per dataset) |
|
|
| 1. **Load** raw data (DICOM via `sitk.ImageSeriesReader`, NIfTI via `sitk.ReadImage`, NRRD). |
| 2. **Extract metadata** from headers, CSV files, or DICOM tags. |
| 3. **Resample** to isotropic spacing (`get_unisize_resampler` in `util.py`). |
| 4. **Clamp intensities** — CT: `[-300, 300]` HU; MRI: per-dataset windows. |
| 5. **Process segmentation labels** with identical resampling (nearest-neighbor). |
| 6. **Validate** image/label dimensions agree (`assert image.GetSize() == label.GetSize()`). |
| 7. **Write** standardized `.nii.gz` and append to `nifti_mappings.json`. |
|
|
| ### Shared `util.py` API |
|
|
| | Function / class | Purpose | |
| |---|---| |
| | `meta_data` | Validates metadata against `config_format.json`; required fields: `Modality`, `OriImg_path`, `Spacing_mm`, `Size`, `Dataset_name`. Normalizes ambiguous terminology via synonym dictionaries. | |
| | `get_unisize_resampler(image)` | Builds a SimpleITK resampler for isotropic spacing; returns `None` if already isotropic. | |
| | `clamp_image(image, lo, hi)` | HU/intensity clamping via `sitk.ClampImageFilter`. | |
|
|
| ## Dependencies |
|
|
| ```bash |
| pip install SimpleITK pandas numpy tqdm openpyxl |
| ``` |
|
|
| (No `requirements.txt` — install manually.) |
|
|
| ## What's Included / Excluded |
|
|
| - ✅ Cleanup scripts, `util.py`, `config_format.json`, demographic CSVs. |
| - ✅ A handful of tiny demo / sample `.nii.gz` files in `PSMA_clean/{sample,demo}/`. |
| - ❌ Raw datasets (download from each dataset's official source). |
| - ❌ Run logs from prior cleanup runs (`*.log`). |
| - ❌ Intermediate test outputs (`MnM2_clean/test/`). |
|
|
| ## License |
|
|
| MIT — see project root. |
|
|
