init

.gitignore

@@ -0,0 +1,62 @@
+# --- site configuration (never commit: contains absolute paths of your machine/cluster) ---------
+# config.hpc.yaml, config.local.yaml, ... all stay local; only the example is tracked.
+config*.yaml
+config*.yml
+!config.example.yaml
+jobs/site.env
+*.sbatch
+!*.sbatch.example
+.env
+
+# --- patient data and anything derived from it --------------------------------------------------
+# Nothing that could carry patient information belongs in this repository. If you ever need to
+# commit a small synthetic file, add it explicitly with `git add -f`.
+*.nii
+*.nii.gz
+*.dcm
+*.dicom
+*.csv
+*.tsv
+*.xlsx
+*.xls
+data/
+nifti/
+synthetic_data/
+tools/rsync_test_folders.py
+
+# --- run artefacts -----------------------------------------------------------------------------
+outputs/
+*.pth
+*.pt
+*.ckpt
+*.pkl
+*.png
+*.pdf
+*.svg
+wandb/
+lightning_logs/
+output_files/
+slurm-*.out
+
+# --- python ------------------------------------------------------------------------------------
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.eggs/
+.venv/
+venv/
+env/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+.ipynb_checkpoints/
+
+# --- editors / os ------------------------------------------------------------------------------
+.idea/
+.vscode/
+*.swp
+.DS_Store

README.md

@@ -1,3 +1,214 @@
----
-license: apache-2.0
----
+# auto_detect_breast_mri
+
+Malignancy classification on breast MRI, comparing an **abbreviated protocol (AP)** against the
+**full dynamic protocol (FDP)**. The importable package is `auto_detect_breast_mri`; every command
+line entry point lives in `scripts/`.
+
+"AP" and "abrv" throughout this repository means *abbreviated protocol*: the native scan plus the first
+subtraction image, instead of the full dynamic series, which is referred to as "FDP" or "full".
+
+## Getting started
+
+With [uv](https://docs.astral.sh/uv/), which reads `pyproject.toml` and installs the package itself:
+
+```bash
+uv sync                                  # creates .venv from uv.lock, package included
+cp config.example.yaml config.yaml       # config*.yaml is gitignored
+$EDITOR config.yaml                      # fill in your paths
+export AIMRI_CONFIG=$PWD/config.yaml
+uv run python scripts/train/train.py resnet18_abrv -b 2 -n 1 -c 0 -d 0 -t -e
+```
+
+With conda or plain pip instead:
+
+```bash
+pip install -r requirements.txt          # or: conda env create -f ai_mri.yml
+pip install -e . --no-deps               # makes `import auto_detect_breast_mri` work
+cp config.example.yaml config.yaml
+$EDITOR config.yaml
+export AIMRI_CONFIG=$PWD/config.yaml
+```
+
+Both routes read the same two files: dependencies come from `requirements.txt` and the version from
+`auto_detect_breast_mri/__init__.py`, which `pyproject.toml` pulls in dynamically.
+
+On a GPU cluster, install the torch build that matches the site's CUDA before `uv sync` (or point uv
+at the matching index), otherwise the default PyPI wheel decides which CUDA runtime you get.
+
+The editable install is what lets the entry points under `scripts/` find the package: Python puts
+the script's own directory on `sys.path`, never the working directory, so without it every script
+fails with `ModuleNotFoundError: No module named 'auto_detect_breast_mri'`. `--no-deps` keeps pip
+from reinstalling torch and friends over an environment that already works. `export PYTHONPATH=$PWD`
+from the repository root does the same for a single shell, without installing anything; that is what
+`jobs/cross_validation.sbatch.example` does.
+
+Everything site specific — paths, cluster account, wandb project, CSV column names — lives in that
+YAML file, never in the code. Command line arguments always take precedence over it, so an
+invocation that passes all paths explicitly needs no config file at all:
+
+```bash
+python scripts/train/train.py resnet18_abrv <data_root> <metadata_file> <split_root> -b 32 -n 60 -c 0 -d 0 -t -e
+python scripts/train/train.py resnet18_abrv -b 32 -n 60 -c 0 -d 0 -t -e     # same, paths from the config
+```
+
+`$AIMRI_CONFIG` may point anywhere, so several site configs can live side by side (for example
+`config.hpc.yaml` for the cluster and `config.local.yaml` for a workstation). Only `config.yaml` and
+`config.yml` in the repository root are found automatically.
+
+### Try it without patient data
+
+```bash
+python tools/make_synthetic_dataset.py --out synthetic_data --masks
+export AIMRI_CONFIG=synthetic_data/config.yaml
+python scripts/train/train.py resnet18_abrv -b 2 -n 1 -c 0 -d 0 -t -e
+```
+
+This writes eight synthetic examinations in **both** supported folder layouts, a matching metadata
+export and split files, plus a ready to use config. The volumes are random noise with a
+label-dependent blob, so a model can actually overfit them. CI runs the tests against this same
+dataset.
+
+## Data contract
+
+### Image folders
+
+Two layouts are supported and may be mixed under one `data_root` (`data_root` may also be a comma
+separated list of roots):
+
+**Bilateral** — one folder per examination, side encoded in the file name:
+
+```
+<data_root>/<examination_id>/
+    L-Dyn_0.nii.gz   L-Sub_1.nii.gz   L-Sub_2.nii.gz  ...  L-T2.nii.gz
+    R-Dyn_0.nii.gz   R-Sub_1.nii.gz   ...
+    L-mask_breast_nn.nii.gz   R-mask_breast_nn.nii.gz     # only for segmentation/cropping
+```
+
+**Unilateral** — one folder per breast, side in the folder name, `Pre`/`Post_X` file names:
+
+```
+<data_root>/<examination_id>_left/
+    Pre.nii.gz   Post_1.nii.gz   Sub_1.nii.gz  ...  T2.nii.gz
+    mask_breast_nn.nii.gz                                  # only for segmentation/cropping
+<data_root>/<examination_id>_right/
+    ...
+```
+
+`Pre` is mapped onto `Dyn_0` and `Post_X` onto `Dyn_X`, so both layouts end up with the same
+canonical sequence names. A folder name ending in `01` is also matched against the ID without those
+two digits, which some exports append.
+
+### Sequence and protocol names
+
+Canonical sequence names: `Dyn_0` (native), `Dyn_1` … `Dyn_4` (post contrast), `Sub_1` … `Sub_4`
+(subtraction, i.e. `Dyn_X` − `Dyn_0`) and `T2`.
+
+| protocol        | expands to                                        |
+|-----------------|---------------------------------------------------|
+| `abbreviated`   | `Dyn_0`, `Sub_1`                                  |
+| `full`          | `Dyn_0`, `Sub_1`, `Sub_2`, `Sub_3`, `Sub_4`, `T2` |
+| `sub`           | `Sub_1`                                           |
+| a list of names | exactly those, in the given order                 |
+
+`Dyn_0` must come first when present; the model's input channel count equals the number of
+sequences. Model names encode the protocol: `*_abrv` → abbreviated, `*_full` → full, `*_sub` → sub.
+
+### Metadata export
+
+One row per examination, given as `metadata_file` (`.csv` or `.xlsx`), with per-side label columns:
+
+| exam_id | exam_id | pat_id    | Malign_Left | Malign_Right |
+|---------|---------|-----------|-------------|--------------|
+| 10001   | 10001   | 90001     | 0           | 1            |
+
+### Split files
+
+One row per breast, under `split_root`, one folder per outer fold. `<k>` is the outer fold, `<s>` the
+inner (validation) fold; the test set has no inner fold:
+
+```
+<split_root>/fold<k>/stratified_training_set-f<k>-<s>.csv
+<split_root>/fold<k>/stratified_evaluation_set-f<k>-<s>.csv
+<split_root>/fold<k>/stratified_test_set-f<k>.csv
+<split_root>/fold<k>/stratified_training_set-f<k>-<s>_frac0.25.csv   # training fractions
+```
+
+| exam_id | pat_id    | Side  | Malign |
+|---------|-----------|-------|--------|
+| 10001   | 90001     | Left  | 0      |
+| 10001   | 90001     | Right | 1      |
+
+The **first column is read positionally** as the examination ID and must match the image folder
+names. Splits are grouped by patient ID, so no patient appears in two folds.
+
+### Column names are configurable
+
+The column names above are the defaults of the original UKA export. Point the code at a differently
+named export with the `columns:` block of your config file — no code change needed:
+
+```yaml
+columns:
+  examination_id: StudyID            # in the split files
+  metadata_examination_id: StudyID   # in the metadata export
+  patient_id: PatientID
+  side: Side
+  criterion: Cancer
+  criterion_left: Cancer_Left
+  criterion_right: Cancer_Right
+  side_left: L
+  side_right: R
+```
+
+## Pipeline
+
+1. **Preprocess** — crop with the trained UNet and split each volume into left/right halves:
+   `python -m auto_detect_breast_mri.preprocessing.split_breasts --unet-path <unet.pth>`
+   (raw DICOM first goes through `auto_detect_breast_mri.preprocessing.anonymize_dicom`).
+2. **Split** — patient-grouped, label-stratified folds plus nested training fractions:
+   `python scripts/data_utils/make_splits.py` and `python scripts/data_utils/make_split_fractions.py`
+   (verify the nesting with `python tools/check_split_fractions.py`).
+3. **Train** — one model per fold: `python scripts/train/train.py <model_name> -c <fold> -d <subfold> -t -e` 
+   On SLURM: `python jobs/run_trainings.py --nets resnet18_abrv --folds 0 1 2 3 4 --dry-run`
+4. **Compare protocols** — abbreviated against full, with DeLong p-value and GradCAM heatmaps:
+   `python scripts/validation/compare_protocols.py <model_name> ... -o <checkpoint pattern>` # checkpoint pattern will be formatted with model_key and fold, so use something like <your_pretrained_models_root>/{}_fold={}<further_filename_parts>.pth  depending on where you have model_key and fold in your filename
+5. **Non-inferiority** — out-of-fold predictions, then the clustered bootstrap:
+   `python scripts/validation/predict_oof.py resnet18 <data_root> <metadata.csv> <split_files_root> -c 0 -m '<path_to_pretrained_models_root>/{model_key}_fold={fold}_subfold=0_frac=0.05_FINAL.pth'` then 
+   `python scripts/validation/analyze_noninferiority.py <output_root_from_predict_oof> <path to metadata.csv> --bca --skip_secondary` 
+6. **Per indication subgroups** — exploratory, on the same out-of-fold predictions:
+   `python scripts/validation/analyze_subgroups.py <output_root_from_predict_oof> <path to metadata.csv>`
+   Reports per indication the AUC of both protocols and the paired AP minus FDP difference with a
+   patient-clustered interval. Give the codes readable names with an `indication_labels:` block in
+   the site config. Exploratory only: the subgroups are small, nothing is adjusted for multiplicity
+   across them, and no margin is tested.
+
+## Layout
+
+```
+auto_detect_breast_mri/
+├── config.py            site configuration ($AIMRI_CONFIG), path and column resolution
+├── data/                datasets, transforms, split generation, dataloaders, metadata/NIfTI IO
+├── models/              ResNets, the cropping UNet, checkpoint IO
+├── training/            training and evaluation loops, LR schedules, shared CLI
+├── evaluation/          metrics, GradCAM, clustered bootstrap, dataset statistics
+└── preprocessing/       DICOM anonymisation, cropping and left/right splitting
+scripts/                 every command line entry point, verb-named
+├── data_utils/          splits, metadata fixes, label checks, dataloader/transform inspection
+├── train/               training entry points (classifier, pretrained baseline, cropping UNet)
+└── validation/          inference, evaluation, non-inferiority analysis, plots and heatmaps
+jobs/                    SLURM templates and launchers (copy the .example files)
+tools/                   synthetic dataset generator, repository checks
+tests/                   unit tests only
+```
+
+## Keeping patient data out of the repository
+
+- `config*.yaml` (except the example), `jobs/site.env` and `*.sbatch` are gitignored: they are the
+  only place where absolute paths, cluster accounts and personal directories belong.
+- `.gitignore` also excludes NIfTI/DICOM/CSV/XLSX files, checkpoints, plots and `wandb/`.
+- `python tools/check_no_private_paths.py` fails on absolute paths, cluster accounts, personal names
+  and bare 7–10 digit numbers (which could be examination IDs). Install it as a pre-commit hook with
+  `pip install pre-commit && pre-commit install`; CI runs it on every push.
+- Weights & Biases: run configs and names are visible to everyone with access to the project, and
+  heatmap captions contain patient keys. Set `wandb_mode: offline` (or `disabled`) in your config
+  when that is not acceptable.
+

auto_detect_breast_mri/__init__.py

@@ -0,0 +1 @@
+__version__ = '0.0.1'

auto_detect_breast_mri/config.py

@@ -0,0 +1,244 @@
+"""
+Site configuration: everything that differs between machines, cluster accounts and metadata exports.
+
+Nothing in this repository may contain an absolute path, a cluster account, a user name or a column
+name of a specific metadata export. Those values live in a YAML file that is *not* under version
+control. `config.example.yaml` documents the shape; copy it and fill it in:
+
+    cp config.example.yaml config.yaml     # config*.yaml is gitignored
+    export AIMRI_CONFIG=/path/to/config.yaml
+
+Resolution order for the config file:
+  1. the path given to `load_config()`
+  2. $AIMRI_CONFIG
+  3. config.yaml / config.yml next to this repository's root
+  4. no config at all -- every value must then be given on the command line
+
+Command line arguments always win over the config file, so any invocation that worked before a
+config file existed keeps working unchanged. Use `resolve_path()` for that precedence:
+
+    path_base = resolve_path(args.data_path, 'data_root', 'root folder of the NIfTI data')
+
+Column names of the label/split CSVs are configurable in the same file under `columns:`; the
+defaults reproduce the original UKA export, so behaviour is unchanged when nothing is configured.
+"""
+
+from __future__ import annotations   # keeps the config importable on interpreters before 3.10
+
+import os
+from pathlib import Path
+
+CONFIG_ENV_VAR = "AIMRI_CONFIG"
+DEFAULT_CONFIG_NAMES = ("config.yaml", "config.yml")
+REPO_ROOT = Path(__file__).resolve().parent.parent  # the checkout, not the package
+
+# Path keys understood in the config file. Values are documented in config.example.yaml.
+PATH_KEYS = ("data_root", "metadata_file", "split_root", "output_root", "model_root")
+
+# Column names of the metadata and split CSVs. The defaults are the original UKA export's names;
+# override any of them in the config file to point the code at a differently named export.
+DEFAULT_COLUMNS = {
+    "examination_id": "AnforderungsNr",           # one MRI examination, in the split CSVs
+    "metadata_examination_id": "AnforderungsNrE",  # the same ID in the metadata export
+    "patient_id": "PNR",                          # patient, i.e. the grouping key for splitting
+    "side": "Side",                               # breast side in the split CSVs
+    "metadata_side": "Seite",                     # breast side in the metadata export
+    "criterion": "Malign",                        # training label, one row per breast
+    "criterion_left": "Malign_Left",              # per-side label in the wide metadata export
+    "criterion_right": "Malign_Right",
+    "side_left": "Left",                          # values found in the side column
+    "side_right": "Right",
+    "indication_left": "IndicationType_Left",     # per-side indication code in the metadata export
+    "indication_right": "IndicationType_Right",
+}
+
+# Meaning of the indication codes, used to label the subgroups of the per-indication analysis.
+# Codes are compared as strings after stripping a trailing '.0', so both 1 and 1.0 map onto "1".
+# The site config overrides this under `indication_labels:`; unmapped codes keep their raw value.
+DEFAULT_INDICATION_LABELS = {}
+
+DEFAULT_WANDB = {
+    "wandb_project": None,
+    "wandb_entity": None,
+    "wandb_mode": None,   # 'online' | 'offline' | 'disabled'; None leaves wandb's own default
+}
+
+
+class ConfigError(RuntimeError):
+    """Raised when a config file exists but cannot be used."""
+
+
+class Config:
+    """Read-only view on the site configuration. Missing values are None rather than an error."""
+
+    def __init__(self, values: dict | None = None, source: Path | None = None):
+        self._values = dict(values or {})
+        self.source = source
+
+        columns = dict(DEFAULT_COLUMNS)
+        configured = self._values.get("columns") or {}
+        if not isinstance(configured, dict):
+            raise ConfigError(f"'columns' in {source} must be a mapping, got {type(configured).__name__}")
+        unknown = set(configured) - set(DEFAULT_COLUMNS)
+        if unknown:
+            raise ConfigError(f"Unknown column keys in {source}: {sorted(unknown)}. "
+                              f"Known keys: {sorted(DEFAULT_COLUMNS)}")
+        columns.update({key: value for key, value in configured.items() if value})
+        self._columns = columns
+
+    def get(self, key: str, default=None):
+        """Value of a top level config key, or `default` if it is absent or empty."""
+        value = self._values.get(key)
+        return default if value in (None, "") else value
+
+    def path(self, key: str, default=None):
+        """Value of a path key as str. Unknown keys are a programming error, not a config error."""
+        if key not in PATH_KEYS:
+            raise KeyError(f"Unknown path key '{key}'. Known keys: {PATH_KEYS}")
+        value = self.get(key, default)
+        return str(value) if value is not None else None
+
+    def column(self, name: str) -> str:
+        """Configured name of a metadata/split CSV column (see DEFAULT_COLUMNS)."""
+        if name not in DEFAULT_COLUMNS:
+            raise KeyError(f"Unknown column '{name}'. Known columns: {sorted(DEFAULT_COLUMNS)}")
+        return self._columns[name]
+
+    @property
+    def columns(self) -> dict:
+        return dict(self._columns)
+
+    @property
+    def indication_labels(self) -> dict:
+        """Code -> readable name for the indication subgroups, keys normalised to strings."""
+        labels = dict(DEFAULT_INDICATION_LABELS)
+        configured = self._values.get("indication_labels") or {}
+        if not isinstance(configured, dict):
+            raise ConfigError(f"'indication_labels' in {self.source} must be a mapping, "
+                              f"got {type(configured).__name__}")
+        labels.update(configured)
+        return {normalise_indication_code(key): str(value) for key, value in labels.items()}
+
+    def wandb(self, key: str, default=None):
+        """Value of a wandb key ('wandb_project', 'wandb_entity', 'wandb_mode')."""
+        if key not in DEFAULT_WANDB:
+            raise KeyError(f"Unknown wandb key '{key}'. Known keys: {sorted(DEFAULT_WANDB)}")
+        return self.get(key, default)
+
+    def wandb_init_kwargs(self, **overrides) -> dict:
+        """
+        Keyword arguments for `wandb.init()` that carry no site information beyond what is
+        configured. Keys whose value is None are omitted so wandb applies its own default.
+        """
+        kwargs = {"project": self.wandb("wandb_project"),
+                  "entity": self.wandb("wandb_entity"),
+                  "mode": self.wandb("wandb_mode")}
+        kwargs.update(overrides)
+        return {key: value for key, value in kwargs.items() if value is not None}
+
+    def __repr__(self):
+        return f"Config(source={self.source}, keys={sorted(self._values)})"
+
+
+def config_path(explicit: str | os.PathLike | None = None) -> Path | None:
+    """The config file that would be used, or None if there is none."""
+    if explicit:
+        return Path(explicit)
+    from_env = os.environ.get(CONFIG_ENV_VAR)
+    if from_env:
+        return Path(from_env)
+    for name in DEFAULT_CONFIG_NAMES:
+        candidate = REPO_ROOT / name
+        if candidate.exists():
+            return candidate
+    return None
+
+
+def load_config(explicit: str | os.PathLike | None = None) -> Config:
+    """
+    Load the site configuration. Returns an empty Config when no config file is present, so the
+    repository stays usable with command line arguments only.
+    """
+    path = config_path(explicit)
+    if path is None:
+        return Config()
+    if not path.exists():
+        raise ConfigError(f"Config file {path} does not exist "
+                          f"(set ${CONFIG_ENV_VAR} to an existing file or unset it).")
+    try:
+        import yaml
+    except ImportError as exc:  # pragma: no cover - depends on the environment
+        raise ConfigError(f"Reading {path} needs PyYAML: pip install pyyaml") from exc
+
+    with open(path) as handle:
+        values = yaml.safe_load(handle) or {}
+    if not isinstance(values, dict):
+        raise ConfigError(f"{path} must contain a mapping at the top level, "
+                          f"got {type(values).__name__}")
+    return Config(values, source=path)
+
+
+_CACHED: Config | None = None
+
+
+def get_config() -> Config:
+    """The process wide site configuration, loaded once."""
+    global _CACHED
+    if _CACHED is None:
+        _CACHED = load_config()
+    return _CACHED
+
+
+def reset_config() -> None:
+    """Forget the cached configuration (for tests that change $AIMRI_CONFIG)."""
+    global _CACHED
+    _CACHED = None
+
+
+def resolve_path(cli_value, key: str, description: str = "", required: bool = True):
+    """
+    Resolve one path: the command line value if given, else the config file value.
+
+    :param cli_value: value coming from argparse, or None
+    :param key: config key to fall back to, one of PATH_KEYS
+    :param description: what the path is, used in the error message
+    :param required: if True, exit with a readable message when neither source provides a value
+    :return: the resolved path as str, or None when not required and unset
+    """
+    if cli_value:
+        return str(cli_value)
+    config = get_config()
+    value = config.path(key)
+    if value:
+        return value
+    if not required:
+        return None
+    where = f"'{key}' in {config.source}" if config.source else f"'{key}' in a config file"
+    raise SystemExit(
+        f"ERROR: no {description or key} given.\n"
+        f"  Pass it on the command line, or set {where}.\n"
+        f"  See config.example.yaml; point ${CONFIG_ENV_VAR} at your copy."
+    )
+
+
+def normalise_indication_code(code) -> str:
+    """
+    The canonical string form of an indication code, so 1, '1', 1.0 and '1.0' are one subgroup.
+    Non numeric codes are kept as their stripped string.
+    """
+    text = str(code).strip()
+    try:
+        number = float(text)
+    except ValueError:
+        return text
+    return str(int(number)) if number.is_integer() else text
+
+
+def indication_labels() -> dict:
+    """Configured code -> readable name mapping for the indication subgroups."""
+    return get_config().indication_labels
+
+
+def column(name: str) -> str:
+    """Configured name of a metadata/split CSV column."""
+    return get_config().column(name)

auto_detect_breast_mri/evaluation/__init__.py

@@ -0,0 +1 @@
+"""Metrics, statistics, GradCAM and the analyses built on predictions."""

auto_detect_breast_mri/evaluation/cluster_bootstrap.py

@@ -0,0 +1,303 @@
+"""
+Fold-stratified, patient-clustered bootstrap for the paired AP vs FDP comparison.
+
+The scores of the five outer folds come from five separately trained models, so their scales are not
+comparable. Pooling all scores into one AUC would silently treat those scale differences as ranking
+information. Everything here therefore computes AUCs *within* a fold and combines the fold level
+differences with weights that are fixed before resampling starts.
+
+Resampling draws whole women with replacement, so all breasts and all repeated examinations of one
+woman enter or leave a replication together, within fold and within patient level outcome stratum.
+
+Limitation: bootstrapping stored out-of-fold scores reflects test sample uncertainty and the patient
+clustering, but not the uncertainty of retraining the models. A fully unconditional bootstrap would
+repeat the whole training pipeline per replication.
+"""
+
+import numpy as np
+
+DEFAULT_MARGINS = (0.02, 0.03, 0.05)
+
+
+def rank_average(values):
+    """
+    Ranks of `values` starting at 1, with tied values receiving their average rank.
+    :param values: 1D array of scores
+    :return: float array of the same length
+    """
+    values = np.asarray(values, dtype=float)
+    order = np.argsort(values, kind='mergesort')
+    sorted_values = values[order]
+    ranks = np.empty(len(values), dtype=float)
+
+    # assign each group of equal values the mean of the ranks it spans
+    group_start = 0
+    for position in range(1, len(sorted_values) + 1):
+        if position == len(sorted_values) or sorted_values[position] != sorted_values[group_start]:
+            mean_rank = 0.5 * (group_start + position + 1)  # ranks group_start+1 .. position
+            ranks[order[group_start:position]] = mean_rank
+            group_start = position
+    return ranks
+
+
+def auc(labels, scores):
+    """
+    Area under the ROC curve via the rank sum of the positives, with ties counted as half.
+    :param labels: binary 1D array, 1 = malignant
+    :param scores: 1D array of scores, higher meaning more suspicious
+    :return: AUC, or nan if one of the two classes is absent
+    """
+    labels = np.asarray(labels)
+    positives = labels == 1
+    n_pos = int(positives.sum())
+    n_neg = int(len(labels) - n_pos)
+    if n_pos == 0 or n_neg == 0:
+        return np.nan
+    ranks = rank_average(scores)
+    return (ranks[positives].sum() - n_pos * (n_pos + 1) / 2.0) / (n_pos * n_neg)
+
+
+def sensitivity_at_specificity(labels, scores, target_specificity):
+    """
+    Sensitivity at the lowest threshold that still reaches `target_specificity` on this same data.
+
+    A case counts as predicted malignant when its score is at or above the threshold, so
+    specificity is the share of benign breasts scoring strictly below it. Among all thresholds
+    meeting the target, the smallest one is chosen, which is the one with the highest sensitivity.
+
+    Note that deriving the threshold on the data it is evaluated on is optimistic; this is only
+    meant for the descriptive secondary analysis.
+    :return: (sensitivity, achieved specificity, threshold), all nan if a class is absent
+    """
+    labels = np.asarray(labels)
+    scores = np.asarray(scores, dtype=float)
+    negatives = np.sort(scores[labels == 0])
+    positives = scores[labels == 1]
+    if len(negatives) == 0 or len(positives) == 0:
+        return np.nan, np.nan, np.nan
+
+    # at least this many benign breasts have to fall strictly below the threshold
+    needed = int(np.ceil(target_specificity * len(negatives)))
+    if needed <= 0:
+        threshold = -np.inf
+    else:
+        boundary = negatives[needed - 1]
+        # the smallest score actually observed above that boundary; thresholds in between would
+        # define the same classifier but a larger one would needlessly cost sensitivity
+        ordered = np.sort(scores)
+        position = np.searchsorted(ordered, boundary, side='right')
+        threshold = ordered[position] if position < len(ordered) else np.inf
+
+    achieved_specificity = float(np.mean(negatives < threshold))
+    sensitivity = float(np.mean(positives >= threshold))
+    return sensitivity, achieved_specificity, float(threshold)
+
+
+def specificity_at_sensitivity(labels, scores, target_sensitivity):
+    """
+    Specificity at the highest threshold that still reaches `target_sensitivity`, the mirror image
+    of sensitivity_at_specificity.
+
+    A case counts as predicted malignant when its score is at or above the threshold, so sensitivity
+    is the share of malignant breasts scoring at or above it. Among all thresholds meeting the
+    target, the largest one is chosen, which is the one with the highest specificity.
+
+    Deriving the threshold on the data it is evaluated on is optimistic, the same caveat as above.
+    :return: (specificity, achieved sensitivity, threshold), all nan if a class is absent
+    """
+    labels = np.asarray(labels)
+    scores = np.asarray(scores, dtype=float)
+    negatives = scores[labels == 0]
+    positives = np.sort(scores[labels == 1])
+    if len(negatives) == 0 or len(positives) == 0:
+        return np.nan, np.nan, np.nan
+
+    # at least this many malignant breasts have to lie at or above the threshold
+    needed = int(np.ceil(target_sensitivity * len(positives)))
+    threshold = -np.inf if needed <= 0 else positives[len(positives) - needed]
+
+    achieved_sensitivity = float(np.mean(positives >= threshold))
+    specificity = float(np.mean(negatives < threshold))
+    return specificity, achieved_sensitivity, float(threshold)
+
+
+class FoldClusterBootstrap:
+    """
+    Resampling machinery for one out-of-fold table.
+
+    :param folds: 1D array assigning each row to an outer fold
+    :param patients: 1D array assigning each row to a woman
+    :param labels: binary 1D array, 1 = malignant
+    :param stratify_by_outcome: bool
+        if True, women are resampled separately among those with and those without any malignant
+        breast in that fold, keeping the class distribution stable across replications
+    """
+
+    def __init__(self, folds, patients, labels, stratify_by_outcome=True):
+        self.folds = np.asarray(folds)
+        self.patients = np.asarray(patients)
+        self.labels = np.asarray(labels)
+        self.stratify_by_outcome = stratify_by_outcome
+        self.fold_ids = np.unique(self.folds)
+
+        # per fold: the row indices of every woman, split into the two outcome strata
+        self.strata_of_fold = {}
+        for fold in self.fold_ids:
+            fold_rows = np.flatnonzero(self.folds == fold)
+            rows_of_patient = {}
+            for row in fold_rows:
+                rows_of_patient.setdefault(self.patients[row], []).append(row)
+
+            strata = {0: [], 1: []}
+            for patient, rows in rows_of_patient.items():
+                rows = np.asarray(rows)
+                stratum = 1 if self.labels[rows].max() == 1 else 0
+                strata[stratum if self.stratify_by_outcome else 0].append(rows)
+            self.strata_of_fold[fold] = [strata[0], strata[1]]
+
+    def fold_weights(self, weighting='pairs'):
+        """
+        Weights used to combine the fold level differences. They are computed once on the observed
+        data and then held fixed across all replications, so they carry no replication specific
+        information.
+        :param weighting: 'pairs' for w_k proportional to positives x negatives, 'cases' for the
+            number of breasts in the fold, or 'equal'
+        :return: dict fold -> weight, summing to 1
+        """
+        weights = {}
+        for fold in self.fold_ids:
+            fold_labels = self.labels[self.folds == fold]
+            n_pos = int((fold_labels == 1).sum())
+            n_neg = int((fold_labels == 0).sum())
+            if weighting == 'pairs':
+                weights[fold] = float(n_pos * n_neg)
+            elif weighting == 'cases':
+                weights[fold] = float(n_pos + n_neg)
+            elif weighting == 'equal':
+                weights[fold] = 1.0
+            else:
+                raise ValueError(f"Unknown weighting '{weighting}'. Use 'pairs', 'cases' or 'equal'.")
+        total = sum(weights.values())
+        return {fold: weight / total for fold, weight in weights.items()}
+
+    def observed_rows(self):
+        """Row indices per fold of the unresampled data."""
+        return {fold: np.flatnonzero(self.folds == fold) for fold in self.fold_ids}
+
+    def resample_rows(self, rng):
+        """
+        Draw one bootstrap replication: women with replacement, within fold and outcome stratum.
+        :return: dict fold -> row indices of the replication (a woman drawn twice contributes all
+            of her breasts twice)
+        """
+        rows_of_fold = {}
+        for fold in self.fold_ids:
+            drawn = []
+            for stratum in self.strata_of_fold[fold]:
+                if not len(stratum):
+                    continue
+                picks = rng.integers(0, len(stratum), size=len(stratum))
+                drawn.extend(stratum[pick] for pick in picks)
+            rows_of_fold[fold] = np.concatenate(drawn) if drawn else np.empty(0, dtype=int)
+        return rows_of_fold
+
+
+def weighted_fold_difference(rows_of_fold, labels, score_a, score_b, weights):
+    """
+    Difference of the two AUCs per fold, combined with the fixed weights.
+    Folds in which an AUC is undefined are dropped and the weights of the remaining folds are
+    renormalized, so a replication is only lost if every fold degenerates.
+    :return: (combined difference, dict fold -> (auc_a, auc_b))
+    """
+    per_fold = {}
+    total_weight = 0.0
+    combined = 0.0
+    for fold, rows in rows_of_fold.items():
+        fold_labels = labels[rows]
+        auc_a = auc(fold_labels, score_a[rows])
+        auc_b = auc(fold_labels, score_b[rows])
+        per_fold[fold] = (auc_a, auc_b)
+        if np.isnan(auc_a) or np.isnan(auc_b):
+            continue
+        combined += weights[fold] * (auc_a - auc_b)
+        total_weight += weights[fold]
+    if total_weight == 0:
+        return np.nan, per_fold
+    return combined / total_weight, per_fold
+
+
+def percentile_interval(replications, confidence):
+    """
+    Two sided percentile interval.
+    :param replications: 1D array of bootstrap statistics
+    :param confidence: e.g. 0.95 or 0.975
+    :return: (lower, upper)
+    """
+    replications = np.asarray(replications)
+    replications = replications[~np.isnan(replications)]
+    alpha = (1.0 - confidence) / 2.0
+    return (float(np.quantile(replications, alpha)),
+            float(np.quantile(replications, 1.0 - alpha)))
+
+
+def bca_interval(replications, observed, jackknife_values, confidence):
+    """
+    Bias corrected and accelerated interval, to be used as a prespecified sensitivity analysis only.
+    :param replications: 1D array of bootstrap statistics
+    :param observed: the statistic on the original data
+    :param jackknife_values: statistic with one woman left out at a time
+    :param confidence: e.g. 0.95
+    :return: (lower, upper)
+    """
+    from scipy.stats import norm
+
+    replications = np.asarray(replications)
+    replications = replications[~np.isnan(replications)]
+    jackknife_values = np.asarray(jackknife_values)
+    jackknife_values = jackknife_values[~np.isnan(jackknife_values)]
+
+    proportion_below = np.mean(replications < observed)
+    if proportion_below <= 0 or proportion_below >= 1:
+        return percentile_interval(replications, confidence)
+    bias_correction = norm.ppf(proportion_below)
+
+    centred = jackknife_values.mean() - jackknife_values
+    denominator = 6.0 * (np.sum(centred ** 2) ** 1.5)
+    acceleration = 0.0 if denominator == 0 else float(np.sum(centred ** 3) / denominator)
+
+    alpha = (1.0 - confidence) / 2.0
+    bounds = []
+    for tail in (alpha, 1.0 - alpha):
+        z = norm.ppf(tail)
+        adjusted = bias_correction + (bias_correction + z) / (1.0 - acceleration * (bias_correction + z))
+        bounds.append(float(np.quantile(replications, norm.cdf(adjusted))))
+    return tuple(bounds)
+
+
+def noninferiority_p_value(replications, margin):
+    """
+    One sided bootstrap p value for H0: delta <= -margin against H1: delta > -margin.
+    Uses the same percentile scale as the reported intervals, so a two sided 95% interval whose
+    lower bound exceeds -margin corresponds to p < 0.025.
+    :return: p value in (0, 1]
+    """
+    replications = np.asarray(replications)
+    replications = replications[~np.isnan(replications)]
+    return float((np.sum(replications <= -margin) + 1) / (len(replications) + 1))
+
+
+def holm_adjust(p_values):
+    """
+    Holm step down adjustment.
+    :param p_values: dict name -> raw p value
+    :return: dict name -> adjusted p value
+    """
+    ordered = sorted(p_values.items(), key=lambda item: item[1])
+    n = len(ordered)
+    adjusted = {}
+    running_max = 0.0
+    for rank, (name, p_value) in enumerate(ordered):
+        candidate = min(1.0, (n - rank) * p_value)
+        running_max = max(running_max, candidate)
+        adjusted[name] = running_max
+    return adjusted

auto_detect_breast_mri/evaluation/dataset_stats.py

@@ -0,0 +1,238 @@
+"""Descriptive statistics and plots of a dataset and of the generated splits."""
+
+import matplotlib.pyplot as plt
+import pandas as pd
+
+from auto_detect_breast_mri.config import column
+
+
+def analyze_dataset(metadata_cleaned: str, llm_metadata: str):
+    print(metadata_cleaned)
+    print(llm_metadata)
+    data_meta_id_key = column('examination_id')
+    if metadata_cleaned.endswith('.xlsx'):
+        data = pd.read_excel(metadata_cleaned)
+    elif metadata_cleaned.endswith('.csv'):
+        data = pd.read_csv(metadata_cleaned)
+    else:
+        raise ValueError("DataSplitter: Don't know how to read this file: " + metadata_cleaned)
+
+    meta_id_key = column('examination_id')
+    if llm_metadata.endswith('.xlsx'):
+        llm_metadata = pd.read_excel(llm_metadata)
+    elif llm_metadata.endswith('.csv'):
+        llm_metadata = pd.read_csv(llm_metadata)
+    else:
+        raise ValueError("DataSplitter: Don't know how to read this file: " + llm_metadata)
+    # only use data from one fold (otherwise redundance)
+
+    llm_metadata = llm_metadata[llm_metadata['Fold'] == 0]
+
+    additional_metadata = data[data[data_meta_id_key].isin(llm_metadata[meta_id_key])]
+    if len(additional_metadata) == 0:
+        # trim last two digits as this file contains all IDs with appended 01
+        trimmed_meta = data[data_meta_id_key].astype(str).str[:-2].astype('int64')
+        data[data_meta_id_key] = trimmed_meta
+        additional_metadata = data[data[data_meta_id_key].isin(llm_metadata[meta_id_key])]
+
+    print(f"Number of patients in dataset: {len(data[column('patient_id')].unique())}")
+    print("Len of nan PNRs: " + str(len(data[data[column('patient_id')] != data[column('patient_id')]])))
+    print("\nNumber of patients per split:")
+
+    print(f"\nNumber of unilateral examinations: {len(data[meta_id_key].unique())}")
+    '''split_groups = llm_metadata.groupby(['Split']).count()
+    print("\nNumber of unilateral examinations per split:")
+    print(split_groups[meta_id_key])'''
+
+    print("\nBIRADS distribution over the whole dataset:")
+    birads_groups = llm_metadata.groupby("BIRADS").count()
+    print(birads_groups[meta_id_key]/(len(llm_metadata)))
+    print("BIRADS absolute numbers:")
+    print(birads_groups[meta_id_key])
+
+    # number of patients
+    if 'Menopause' in additional_metadata.keys():
+        print("\nMenopausal status over the whole dataset:")
+        menopause_group = additional_metadata.groupby('Menopause').count()
+        print(menopause_group[data_meta_id_key])
+        print(f"Unknown = {len(additional_metadata[additional_metadata['Menopause'] != additional_metadata['Menopause']])}")
+
+    if 'IndicationType_Left' in additional_metadata.keys() and 'IndicationType_Right' in additional_metadata.keys():
+        print("\nIndication per patient over the whole dataset")
+        indications = pd.melt(additional_metadata, id_vars=[data_meta_id_key], value_vars=['IndicationType_Left', 'IndicationType_Right'],
+                                    var_name=column('side'), value_name='Indication')
+        indications[column('side')] = indications[column('side')].str.extract(r'_(\w+)')
+        print(indications.groupby('Indication').count()[data_meta_id_key])
+
+    if 'BIRADS_Left' in data.keys() and 'BIRADS_Right' in data.keys():
+        birads = pd.melt(data, id_vars=[data_meta_id_key, column('patient_id'), ], value_vars=['BIRADS_Left', 'BIRADS_Right'], var_name=column('side'),
+                         value_name='BIRADS')
+        birads[column('side')] = birads[column('side')].str.extract(r'_(\w+)')
+        data.groupby([column('patient_id'), 'BIRADS_Left', 'BIRADS_Right']).agg('last')
+        print(f"\nTotal number of patients at beginning {len(data[column('patient_id')].unique())}")
+        certain_left = data[data['MalignCertainty_Left'] > 19]
+        certain_both = certain_left[certain_left['MalignCertainty_Right'] > 19]
+        print(f"\nNumber of patients with certainty <= 20 {len(data[column('patient_id')].unique()) - len(certain_both[column('patient_id')].unique())}")
+
+    ''' 
+    mean_age = grouped['Age'].mean()
+    age_min = grouped['Age'].min()
+    age_max = grouped['Age'].max()
+    print("Mean Age: ________________________")
+    print(mean_age)
+    print("Minimum Age: ________________________")
+    print(age_min)
+    print("Maximum Age: ________________________")
+    print(age_max)
+    '''
+
+
+def analyze_split(metadata: str, split_files: [str]):
+    '''
+    Analyze a split defined by csv file with the metadata defined in given metadata file
+    :param metadata: path to csv file with metadata. 'AnforderungsNrE' is a required key.
+    :param split_file: path to csv file about the according split.
+    Required keys are the configured examination ID, patient ID, side and criterion columns.
+    :return:
+    '''
+    print(metadata)
+    print(split_files)
+    data_meta_id_key = column('metadata_examination_id')
+    data = pd.read_csv(metadata)
+
+    split_id_key = column('examination_id')
+    all_split_data = []
+    all_splits_results = {}
+    split_name = {0: 'train', 1: 'eval', 2: 'test'}
+    for i, split_file in enumerate(split_files):
+        if 'Fold0' in data.keys():
+            data = data[data['Fold0'] == split_name[i%3]]
+        split_data = pd.read_csv(split_file)
+        split_results = {}
+
+        split_metadata = data[data[data_meta_id_key].isin(split_data[split_id_key])]
+        if len(split_metadata) == 0:
+            # trim last two digits as this file contains all IDs with appended 01
+            trimmed_meta = data[data_meta_id_key].astype(str).str[:-2].astype('int64')
+            data[data_meta_id_key] = trimmed_meta
+            split_metadata = data[data[data_meta_id_key].isin(split_data[split_id_key])]
+
+        split_id_with_no_metadata = split_data[~ split_data[split_id_key].isin(split_metadata[data_meta_id_key])]
+        print(f"Number of examinations in splitfile {split_file} for which no metadata is found: {len(split_id_with_no_metadata)}")
+        all_split_data.append(split_data)
+        split_id_with_metadata = split_data[split_data[split_id_key].isin(split_metadata[data_meta_id_key])]
+        split_results['Number of available metadata for split'] = len(split_id_with_metadata)
+        split_results['Number of exams without metadata for split'] = len(split_id_with_no_metadata)
+
+
+        split_results['Patient_number'] = len(split_data[column('patient_id')].unique())
+        split_results['Examination_number'] = len(split_data[split_id_key].unique())
+        split_results['Exams without matching metadata'] = len(split_id_with_no_metadata)
+
+        # BIRADS
+        birads = pd.melt(split_metadata, id_vars=[data_meta_id_key],
+                value_vars=['BIRADS_Left', 'BIRADS_Right'],
+                var_name=column('side'), value_name='BIRADS')
+        birads[column('side')] = birads[column('side')].str.extract(r'_(\w+)')
+        birads_groups = birads.groupby("BIRADS").count()
+        split_results['BIRADS distribution'] = birads_groups[data_meta_id_key] / (len(split_data))
+        split_results['BIRADS absolute'] = birads_groups[data_meta_id_key]
+
+        # Malign
+        split_results['Malign %'] = split_data[column('criterion')].sum() / len(split_data[column('criterion')])
+        split_results['Malign absolute'] = {'True': len(split_data[split_data[column('criterion')] == 1]), 'False': len(split_data[split_data[column('criterion')] == 0])}
+
+        j = 0
+        for prev in range(j, i):
+            print("Compare split {} with {}".format(split_files[prev], split_file))
+            prev_split_data = all_split_data[prev]
+            doubles = prev_split_data[prev_split_data[split_id_key].isin(split_data[split_id_key])][split_id_key]
+            split_results[f'PNRs in split {prev} and {i}'] = doubles
+            split_results[f'PNRs COUNT in split {prev} and {i}'] = len(doubles)
+        print(split_results)
+        all_splits_results[split_file.split("cross_validation_sets/")[-1].replace("fold0/stratified_", "")] = split_results
+    return all_splits_results
+
+
+# Function to plot BIRADS distribution
+def plot_birads_distribution2(split_name, split_data):
+    birads_distribution = split_data['BIRADS distribution']
+    birads_absolute = split_data['BIRADS absolute']
+
+    # Convert to pandas Series for easy plotting
+    birads_dist_series = pd.Series(birads_distribution)
+    birads_abs_series = pd.Series(birads_absolute)
+
+    # Plot BIRADS distribution
+    plt.figure(figsize=(12, 5))
+
+    plt.subplot(1, 2, 1)
+    birads_dist_series.plot(kind='bar')
+    plt.title(f'{split_name} - BIRADS Distribution (Percentage)')
+    plt.xlabel('BIRADS')
+    plt.ylabel('Percentage')
+
+    plt.subplot(1, 2, 2)
+    birads_abs_series.plot(kind='bar')
+    plt.title(f'{split_name} - BIRADS Distribution (Absolute)')
+    plt.xlabel('BIRADS')
+    plt.ylabel('Count')
+
+    plt.tight_layout()
+    plt.show()
+
+
+# Function to plot Malignancy distribution
+def plot_malign_distribution(split_name, split_data):
+    malign_percent = split_data['Malign %']
+    malign_absolute = split_data['Malign absolute']
+
+    labels = ['True', 'False']
+    sizes = [malign_absolute['True'], malign_absolute['False']]
+
+    plt.figure(figsize=(6, 6))
+    plt.pie(sizes, labels=labels, autopct='%1.1f%%', startangle=140)
+    plt.title(f'{split_name} - Malignancy Distribution\n(Malign %: {malign_percent:.2%})')
+    plt.show()
+
+
+def plot_birads_distribution(split_name, data):
+    birads_labels = list(data['BIRADS distribution'].keys())
+    birads_values = list(data['BIRADS distribution'].values)
+
+    plt.figure(figsize=(10, 6))
+    plt.bar(birads_labels, birads_values, color='blue', alpha=0.7)
+    plt.xlabel('BIRADS Category')
+    plt.ylabel('Distribution')
+    plt.title(f'{split_name} - BIRADS Distribution')
+    plt.grid(axis='y', linestyle='--', alpha=0.7)
+    plt.show()
+
+
+def plot_birads_absolute(split_name, data):
+    birads_labels = list(data['BIRADS absolute'].keys())
+    birads_values = list(data['BIRADS absolute'].values)
+
+    plt.figure(figsize=(10, 6))
+    plt.bar(birads_labels, birads_values, color='green', alpha=0.7)
+    plt.xlabel('BIRADS Category')
+    plt.ylabel('Absolute Counts')
+    plt.title(f'{split_name} - BIRADS Absolute Counts')
+    plt.grid(axis='y', linestyle='--', alpha=0.7)
+    plt.show()
+
+
+def plot_malignancy(split_name, data):
+    labels = list(data['Malign absolute'].keys())
+    values = list(data['Malign absolute'].values())
+
+    plt.figure(figsize=(6, 6))
+    plt.bar(labels, values, color='red', alpha=0.7)
+    plt.xlabel('Malignancy')
+    plt.ylabel('Counts')
+    plt.title(f'{split_name} - Malignancy Counts')
+    plt.grid(axis='y', linestyle='--', alpha=0.7)
+    plt.show()
+
+
+

auto_detect_breast_mri/evaluation/gradcam.py

@@ -0,0 +1,299 @@
+import logging
+import numpy as np
+import torch
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+logger = logging.getLogger("gradcam")
+
+# Layer types GradCAM can meaningfully hook: convolutions preserve spatial
+# structure directly; the listed norm types are used as a fallback spatial
+# proxy for transformer blocks that have no Conv3d (e.g. Swin stages).
+_CANDIDATE_TYPES = (torch.nn.Conv3d, torch.nn.Conv2d, torch.nn.LayerNorm,
+                    torch.nn.InstanceNorm3d, torch.nn.BatchNorm3d)
+
+
+# --------------------------------------------------------------------------
+# Layout adapters
+#
+# pytorch_grad_cam requires the hooked activations/gradients to be channels-
+# first with the same leading batch as `input_tensor`. render_heatmap always
+# feeds a single 5D volume (1, C, W, H, D) as torchio orders it, so every adapter below normalises
+# an observed activation to (1, C, d, h, w). Anything we cannot map that way is
+# rejected outright rather than fed to GradCAM -- a mis-mapped layout does not
+# raise, it silently yields a meaningless heatmap.
+# --------------------------------------------------------------------------
+
+def _identity(t):
+    return t
+
+
+def _mean_over_leading(t):
+    """(g, C, d, h, w) -> (1, C, d, h, w): average head/group-split activations.
+
+    MViT's attention pooling convs run on (B*num_heads, head_dim, t, h, w);
+    averaging the heads keeps one CAM per sample instead of silently reporting
+    head 0 only.
+    """
+    return t.mean(dim=0, keepdim=True)
+
+
+def _add_depth_axis(t):
+    """(1, C, h, w) -> (1, C, 1, h, w)."""
+    return t.unsqueeze(2)
+
+
+def _slices_to_depth_first(t):
+    """(N, C, h, w) -> (1, C, N, h, w): a per-slice 2D backbone's slice batch is
+    the volume's depth axis."""
+    return t.permute(1, 0, 2, 3).unsqueeze(0)
+
+
+def _slices_to_depth_last(t):
+    """(N, h, w, C) -> (1, C, N, h, w): channels-last per-slice activations."""
+    return t.permute(3, 0, 1, 2).unsqueeze(0)
+
+
+def _channels_last_to_first(t):
+    """(B, d, h, w, C) -> (B, C, d, h, w): MONAI SwinUNETR stages keep the
+    channel axis last."""
+    return t.permute(0, 4, 1, 2, 3)
+
+
+def _layout_transform(module, shape, batch):
+    """Return the reshape_transform mapping `module`'s activation layout to
+    (1, C, d, h, w), or None if the layout is not interpretable.
+
+    `torch.nn.LayerNorm` normalises over the *trailing* dims by construction, so
+    its output is channels-last by definition -- that is a fact about the module,
+    not a guess from the shape. Conv outputs are always channels-first.
+    """
+    ndim = len(shape)
+    channels_last = isinstance(module, torch.nn.LayerNorm)
+    if ndim == 5:
+        if channels_last:
+            # (B, d, h, w, C): three middle dims are an unambiguous spatial grid.
+            return _channels_last_to_first if shape[0] == batch else None
+        return _identity if shape[0] == batch else _mean_over_leading
+    if ndim == 4:
+        if channels_last:
+            # Only a slice-stacked per-slice 2D backbone yields a trustworthy
+            # (N, h, w, C) grid. When the leading dim IS the batch, the two
+            # middle dims of a 4D channels-last tensor are just as likely to be
+            # (heads, tokens) as (h, w) -- MViT's attention norms are exactly
+            # that -- so reject rather than emit a meaningless heatmap.
+            return _slices_to_depth_last if shape[0] != batch else None
+        return _add_depth_axis if shape[0] == batch else _slices_to_depth_first
+    # (B, C) pooled embeddings and (B, tokens, C) sequences carry no recoverable
+    # spatial grid -> not usable as a CAM target.
+    return None
+
+
+def resolve_cam_target(model, model_name, sample_input=None):
+    """Return (target_layer, reshape_transform) for GradCAM.
+
+    If `sample_input` is given, the selection is *empirical*: one lightweight
+    forward+backward pass records which candidate layers GradCAM would actually
+    be able to read, and the deepest usable one is returned. A candidate is
+    usable only if BOTH hold:
+
+      1. its output tensor's `register_hook` fires during backward -- this is
+         exactly how pytorch_grad_cam captures gradients. Checking `.grad` via
+         `retain_grad()` instead is NOT equivalent: nnU-Net's
+         `LeakyReLU(inplace=True)` rebases the norm output's autograd node, so
+         `.grad` is populated while the tensor hook never fires, and GradCAM
+         then sees `grads=None`.
+      2. its activation layout maps onto (1, C, d, h, w) (see _layout_transform).
+         Without this check the deepest connected layer is often a head
+         LayerNorm over a pooled (B, C) embedding or a (B, tokens, C) sequence,
+         which makes GradCAM raise "Invalid grads shape".
+
+    This also keeps multi-task models honest: challenge_2BCN_AIM's SwinUNETR
+    decoder/segmentation head is not on the classification path, so its layers
+    never fire and are skipped.
+
+    Falls back to the static per-architecture heuristic below when no
+    `sample_input` is given or no candidate qualifies.
+    """
+    if sample_input is not None:
+        try:
+            layer, transform = _connected_target_layer(model, sample_input)
+            if layer is not None:
+                return layer, transform
+        except Exception as exc:
+            logger.warning(
+                "GradCAM connectivity probe failed for %s (%s); falling back "
+                "to heuristic target-layer selection.", model_name, exc,
+            )
+    return _heuristic_target_layer(model, model_name), None
+
+
+def get_target_layer(model, model_name, sample_input=None):
+    """Return the layer GradCAM hooks (see resolve_cam_target)."""
+    return resolve_cam_target(model, model_name, sample_input)[0]
+
+
+def _heuristic_target_layer(model, model_name):
+    conv_layers = [m for m in model.modules() if isinstance(m, torch.nn.Conv3d)]
+    if model_name in ("challenge_1DivideAndConquer", "challenge_5pimed") and conv_layers:
+        return conv_layers[-1]
+    if model_name == "challenge_3agaldran":  # MViT: hook the last conv in the temporal reduce / core
+        return conv_layers[-1] if conv_layers else _last_norm(model)
+    if model_name == "challenge_2BCN_AIM":
+        # SwinUNETRMultiTask's classification head only consumes the Swin
+        # ENCODER features (hidden_states_out); the decoder + segmentation
+        # head hung off model.swin_unetr are gradient-disconnected from the
+        # classification logits. Restrict the search to the encoder so we
+        # never return a decoder/seg-head layer here.
+        swin_vit = getattr(getattr(model, "swin_unetr", None), "swinViT", None)
+        if swin_vit is not None:
+            sub_convs = [m for m in swin_vit.modules() if isinstance(m, torch.nn.Conv3d)]
+            return sub_convs[-1] if sub_convs else _last_norm(swin_vit)
+        return _last_norm(model)
+    # challenge_4abmil (ABMIL-Swin, no Conv3d) and any other model -> last conv else last norm
+    if conv_layers:
+        return conv_layers[-1]
+    return _last_norm(model)
+
+
+def _connected_target_layer(model, sample_input):
+    """Probe which candidate layer GradCAM can actually consume.
+
+    Registers forward hooks on every candidate, runs one forward + backward pass
+    (batch size 1, summed logits) while recording each output's shape and
+    whether its gradient hook fired, and returns the deepest candidate (last in
+    `model.modules()` order) that both received gradient and has a usable
+    layout. Hooks are always removed before returning. Returns (None, None) if
+    no candidate qualifies.
+    """
+    candidates = [m for m in model.modules() if isinstance(m, _CANDIDATE_TYPES)]
+    if not candidates:
+        return None, None
+
+    shapes = {}
+    fired = set()
+    hooks = []
+
+    def _make_hook(mod):
+        def _hook(module, inputs, output):
+            tensor = output[0] if isinstance(output, (tuple, list)) else output
+            if not isinstance(tensor, torch.Tensor):
+                return
+            shapes[mod] = tuple(tensor.shape)
+            if tensor.requires_grad:
+                tensor.register_hook(lambda grad, m=mod: fired.add(m))
+        return _hook
+
+    for m in candidates:
+        hooks.append(m.register_forward_hook(_make_hook(m)))
+
+    was_training = model.training
+    single = sample_input[:1] if sample_input.shape[0] > 1 else sample_input
+    try:
+        model.eval()
+        with torch.enable_grad():
+            model.zero_grad(set_to_none=True)
+            output = model(single)
+            logits = output[0] if isinstance(output, (tuple, list)) else output
+            logits.sum().backward()
+    finally:
+        for h in hooks:
+            h.remove()
+        model.train(was_training)
+
+    batch = single.shape[0]
+    for m in reversed(candidates):
+        if m not in fired:
+            continue
+        transform = _layout_transform(m, shapes[m], batch)
+        if transform is not None:
+            return m, transform
+    return None, None
+
+
+def _last_norm(model):
+    norms = [m for m in model.modules()
+             if isinstance(m, (torch.nn.LayerNorm, torch.nn.InstanceNorm3d,
+                               torch.nn.BatchNorm3d))]
+    if not norms:
+        raise RuntimeError("No conv or norm layer found for GradCAM target.")
+    return norms[-1]
+
+
+def select_heatmap_cases(dataset):
+    """Deterministic: sort by UID, first sample per (label, center)."""
+    picks = {}
+    for s in sorted(dataset.samples, key=lambda s: s["uid"]):
+        key = (s["label"], s["center"])
+        if key not in picks:
+            picks[key] = s
+    return list(picks.values())
+
+
+def uka_label_of(dataset, index):
+    """Label of `dataset[index]`, read from the untransformed Subject.
+
+    The label is already resolved from the split CSV when BreastMRISubjects builds
+    its Subjects, so heatmap cases can be picked cheaply, without reading any
+    NIfTI file.
+    """
+    return int(float(dataset.subjects[index]['label']))
+
+
+def uka_key_of(dataset, index):
+    """Patient key ('<PatID><side>') of `dataset[index]`."""
+    return dataset.subjects[index]['path']
+
+
+def select_uka_heatmap_cases(dataset, per_label=1):
+    """Deterministic: sort by patient key, take the first `per_label` samples per label.
+
+    Returns [(index, label), ...] for a BreastMRISubjects. Samples whose label
+    cannot be resolved (e.g. missing row in the split file) are skipped rather
+    than aborting the selection.
+    """
+    order = sorted(range(len(dataset.subjects)), key=lambda i: uka_key_of(dataset, i))
+    picks, counts = [], {}
+    for index in order:
+        try:
+            label = uka_label_of(dataset, index)
+        except Exception as exc:
+            logger.warning("Could not read label of sample %d (%s); skipping.", index, exc)
+            continue
+        if counts.get(label, 0) >= per_label:
+            continue
+        counts[label] = counts.get(label, 0) + 1
+        picks.append((index, label))
+    return picks
+
+
+def render_heatmap(model, model_name, volume, output_png, device=None):
+    # The design requires render_heatmap to NEVER crash the eval run: target-layer
+    # resolution, device transfers, CAM computation, and saving all live inside
+    # this try/except so any failure (e.g. no connected/spatial layer found,
+    # a finicky transformer CAM, an I/O error) is logged and swallowed.
+    try:
+        from pytorch_grad_cam import GradCAM
+        device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+        model = model.to(device).eval()
+        volume = volume.to(device)
+        target_layer, reshape = resolve_cam_target(model, model_name, sample_input=volume)
+        cam = GradCAM(model=model, target_layers=[target_layer],
+                      reshape_transform=reshape)
+        grayscale = cam(input_tensor=volume)  # (1, W, H, D) or (1, W, H)
+        heat = np.asarray(grayscale)[0]
+        # torchio hands over volumes as (C, W, H, D), so the slice axis is the LAST one, not the
+        # first spatial one. Cutting along dim 2 gives a W x D strip (e.g. 256 x 32), which is why
+        # the overlay came out as a squeezed sliver instead of an in-plane image.
+        img_slice = volume[0, 0, :, :, volume.shape[-1] // 2].detach().cpu().numpy()
+        heat_slice = heat[..., heat.shape[-1] // 2] if heat.ndim == 3 else heat
+        fig, ax = plt.subplots()
+        # .T so height runs down the image and width across it, the usual in-plane orientation
+        ax.imshow(img_slice.T, cmap="gray")
+        ax.imshow(heat_slice.T, cmap="jet", alpha=0.4)
+        ax.axis("off")
+        fig.savefig(output_png, bbox_inches="tight")
+        plt.close(fig)
+    except Exception as exc:  # transformer CAMs can be finicky; never crash the eval run
+        logger.warning("GradCAM failed for %s (%s); skipping heatmap.", model_name, exc)

auto_detect_breast_mri/evaluation/metrics.py

@@ -0,0 +1,459 @@
+import os
+import numpy as np
+import pickle
+import pandas as pd
+import pylab as pl
+import scipy.stats
+import torch
+import wandb
+from sklearn.metrics import confusion_matrix, roc_curve, auc
+
+from auto_detect_breast_mri.evaluation.cluster_bootstrap import (sensitivity_at_specificity,
+                                                                 specificity_at_sensitivity)
+
+import matplotlib.pyplot as plt
+
+# The rate held fixed when reading off the other one, see log_operating_points.
+DEFAULT_OPERATING_TARGET = 0.9
+
+
+# Copied from https://github.com/yandexdataschool/roc_comparison/blob/master/compare_auc_delong_xu.py
+
+# AUC comparison adapted from
+# https://github.com/Netflix/vmaf/
+def compute_midrank(x):
+    """Computes midranks.
+    Args:
+       x - a 1D numpy array
+    Returns:
+       array of midranks
+    """
+    J = np.argsort(x)
+    Z = x[J]
+    N = len(x)
+    T = np.zeros(N, dtype=np.float64)
+    i = 0
+    while i < N:
+        j = i
+        while j < N and Z[j] == Z[i]:
+            j += 1
+        T[i:j] = 0.5*(i + j - 1)
+        i = j
+    T2 = np.empty(N, dtype=np.float64)
+    # Note(kazeevn) +1 is due to Python using 0-based indexing
+    # instead of 1-based in the AUC formula in the paper
+    T2[J] = T + 1
+    return T2
+
+def compute_ground_truth_statistics(ground_truth):
+    assert np.array_equal(np.unique(ground_truth), [0, 1])
+    order = (-ground_truth).argsort()
+    label_1_count = int(ground_truth.sum())
+    return order, label_1_count
+
+
+def fastDeLong(predictions_sorted_transposed, label_1_count):
+    """
+    The fast version of DeLong's method for computing the covariance of
+    unadjusted AUC.
+    Args:
+       predictions_sorted_transposed: a 2D numpy.array[n_classifiers, n_examples]
+          sorted such as the examples with label "1" are first
+    Returns:
+       (AUC value, DeLong covariance)
+    Reference:
+     @article{sun2014fast,
+       title={Fast Implementation of DeLong's Algorithm for
+              Comparing the Areas Under Correlated Receiver Operating Characteristic Curves},
+       author={Xu Sun and Weichao Xu},
+       journal={IEEE Signal Processing Letters},
+       volume={21},
+       number={11},
+       pages={1389--1393},
+       year={2014},
+       publisher={IEEE}
+     }
+    """
+    # Short variables are named as they are in the paper
+    m = label_1_count
+    n = predictions_sorted_transposed.shape[1] - m
+    positive_examples = predictions_sorted_transposed[:, :m]
+    negative_examples = predictions_sorted_transposed[:, m:]
+    k = predictions_sorted_transposed.shape[0]
+
+    tx = np.empty([k, m], dtype=np.float64)
+    ty = np.empty([k, n], dtype=np.float64)
+    tz = np.empty([k, m + n], dtype=np.float64)
+    for r in range(k):
+        tx[r, :] = compute_midrank(positive_examples[r, :])
+        ty[r, :] = compute_midrank(negative_examples[r, :])
+        tz[r, :] = compute_midrank(predictions_sorted_transposed[r, :])
+    aucs = tz[:, :m].sum(axis=1) / m / n - float(m + 1.0) / 2.0 / n
+    v01 = (tz[:, :m] - tx[:, :]) / n
+    v10 = 1.0 - (tz[:, m:] - ty[:, :]) / m
+    sx = np.cov(v01)
+    sy = np.cov(v10)
+    delongcov = sx / m + sy / n
+    return aucs, delongcov
+
+
+def calc_pvalue(aucs, sigma):
+    """Computes log(10) of p-values.
+    Args:
+       aucs: 1D array of AUCs
+       sigma: AUC DeLong covariances
+    Returns:
+       pvalue
+    """
+    l = np.array([[1, -1]])
+    z = np.abs(np.diff(aucs)) / np.sqrt(np.dot(np.dot(l, sigma), l.T))
+    return 2 * scipy.stats.norm.sf(z)
+
+
+# modified according to https://github.com/yandexdataschool/roc_comparison/issues/10
+def delong_roc_test(ground_truth, predictions_one, predictions_two):
+    """
+    Computes log(p-value) for hypothesis that two ROC AUCs are different
+    Args:
+       ground_truth: np.array of 0 and 1
+       predictions_one: predictions of the first model,
+          np.array of floats of the probability of being class 1
+       predictions_two: predictions of the second model,
+          np.array of floats of the probability of being class 1
+    """
+    order, label_1_count = compute_ground_truth_statistics(ground_truth.flatten())
+    predictions_sorted_transposed = np.vstack((predictions_one.flatten(), predictions_two.flatten()))[:, order]
+    aucs, delongcov = fastDeLong(predictions_sorted_transposed, label_1_count)
+    return calc_pvalue(aucs, delongcov) 
+
+def _as_numpy(tensor):
+    """Detach a torch tensor into a flat numpy array; pass numpy and lists through."""
+    if isinstance(tensor, torch.Tensor):
+        return tensor.detach().cpu().flatten().numpy()
+    return np.asarray(tensor).flatten()
+
+
+def log_operating_points(y_true, y_scores, target=DEFAULT_OPERATING_TARGET, prefix=''):
+    """
+    Report the two fixed operating points of a score: the sensitivity reached once specificity is
+    held at `target`, and the specificity reached once sensitivity is held at `target`.
+
+    The single number the callers log next to the AUC is taken at the model's own 0.5 threshold,
+    which says little on an imbalanced set and is not comparable between models. Pinning one of the
+    two rates and reading off the other is, which is what this adds.
+
+    The thresholds come from the same data they are evaluated on, so these values are optimistic
+    and descriptive only, exactly like the secondary analysis in analyze_noninferiority.py.
+
+    :param y_true: ground truth binary labels
+    :param y_scores: continuous scores/probabilities of the positive class
+    :param target: the rate that is held fixed, for both directions. Default 0.9.
+    :param prefix: prepended to the wandb keys, e.g. 'Test'
+    :return: dict of the logged values, empty if a class is absent from y_true
+    """
+    labels, scores = _as_numpy(y_true), _as_numpy(y_scores)
+    sens_at, achieved_spec, threshold_spec = sensitivity_at_specificity(labels, scores, target)
+    spec_at, achieved_sens, threshold_sens = specificity_at_sensitivity(labels, scores, target)
+    if np.isnan(sens_at) or np.isnan(spec_at):
+        print(f"Operating points at {target:.0%} need both classes; y_true holds only "
+              f"{sorted(set(labels.tolist()))}.")
+        return {}
+
+    print(f"At specificity {achieved_spec:.4f} (target {target:.0%}, threshold {threshold_spec:.4f}): "
+          f"sensitivity {sens_at:.4f}")
+    print(f"At sensitivity {achieved_sens:.4f} (target {target:.0%}, threshold {threshold_sens:.4f}): "
+          f"specificity {spec_at:.4f}")
+    values = {f"sensitivity_at_specificity{target:g}" + (f"/{prefix}" if prefix else ""): sens_at,
+              f"specificity_at_sensitivity{target:g}" + (f"/{prefix}" if prefix else ""): spec_at,
+              f"threshold_at_specificity{target:g}" + (f"/{prefix}" if prefix else ""): threshold_spec,
+              f"threshold_at_sensitivity{target:g}" + (f"/{prefix}" if prefix else ""): threshold_sens}
+    if wandb.run is not None:
+        wandb.log(values)
+    return values
+
+
+def compute_sensitivity_specificity(y_true, y_pred, y_scores=None,
+                                    target=DEFAULT_OPERATING_TARGET, prefix=''):
+    """
+    Compute sensitivity (true positive rate) and specificity (true negative rate).
+
+    Parameters:
+    - y_true: Ground truth labels (true binary labels).
+    - y_pred: Predicted labels (predicted binary labels).
+    - y_scores: Continuous scores of the positive class. If given, the two fixed operating points
+      are logged in addition (see log_operating_points): sensitivity at specificity `target` and
+      specificity at sensitivity `target`. Without scores only the 0.5 threshold pair is available.
+    - target: the rate held fixed for those operating points. Default 0.9.
+    - prefix: prepended to the wandb keys of those operating points, e.g. 'Test'.
+
+    Returns:
+    - sensitivity: Sensitivity (true positive rate).
+    - specificity: Specificity (true negative rate).
+    """
+    if y_scores is not None:
+        log_operating_points(y_true, y_scores, target=target, prefix=prefix)
+
+    if not torch.all(y_true == y_pred):
+        tn, fp, fn, tp = confusion_matrix(y_true.flatten(), y_pred.flatten()).ravel()
+        sensitivity = tp / (tp + fn) if (tp + fn) != 0 else tp
+        specificity = tn / (tn + fp) if (tn + fp) != 0 else tn
+        return sensitivity, specificity
+    return 1, 1
+
+
+def compute_stratified_metrics(y_true, y_pred, y_scores, metadata,
+                               indication_column='Indication', birads_column='BIRADS'):
+    """Compute overall and subgroup metrics for aligned breast-level predictions.
+
+    ``metadata`` must contain one row per prediction, in the same order as the three
+    prediction arrays. The returned DataFrame has one overall row and one row for
+    every observed indication and BI-RADS category. AUC is ``NaN`` when a group
+    contains only one class, because an ROC curve is undefined in that case.
+    """
+    if not isinstance(metadata, pd.DataFrame):
+        raise TypeError("metadata must be a pandas DataFrame")
+
+    true_values = np.asarray(y_true).reshape(-1)
+    predicted_values = np.asarray(y_pred).reshape(-1)
+    score_values = np.asarray(y_scores).reshape(-1)
+    if not (len(true_values) == len(predicted_values) == len(score_values) == len(metadata)):
+        raise ValueError("y_true, y_pred, y_scores, and metadata must have the same length")
+
+    missing_columns = [name for name in (indication_column, birads_column)
+                       if name not in metadata.columns]
+    if missing_columns:
+        raise ValueError(f"metadata is missing stratification column(s): {missing_columns}")
+
+    def metric_row(scope, category, indices):
+        group_true = true_values[indices].astype(int)
+        group_pred = predicted_values[indices].astype(int)
+        group_scores = score_values[indices].astype(float)
+        sensitivity, specificity = compute_sensitivity_specificity(group_true, group_pred)
+        roc_auc = (float(auc(*roc_curve(group_true, group_scores)[:2]))
+                   if np.unique(group_true).size == 2 else np.nan)
+        return {
+            'scope': scope,
+            'category': category,
+            'n': len(indices),
+            'positives': int(group_true.sum()),
+            'sensitivity': float(sensitivity),
+            'specificity': float(specificity),
+            'auc': roc_auc,
+        }
+
+    rows = [metric_row('overall', 'overall', np.arange(len(true_values)))]
+    for scope, column_name in (('indication', indication_column), ('birads', birads_column)):
+        categories = metadata[column_name].astype('object').where(
+            metadata[column_name].notna(), 'Unknown')
+        for category, group in categories.groupby(categories, sort=True):
+            rows.append(metric_row(scope, category, group.index.to_numpy()))
+    return pd.DataFrame(rows, columns=[
+        'scope', 'category', 'n', 'positives', 'sensitivity', 'specificity', 'auc'])
+
+def compute_roc_curve(y_true, y_scores, plot=False, title_suffix=''):
+    """
+    Compute the ROC curve and AUC (Area Under the Curve).
+
+    Parameters:
+    - y_true: Ground truth labels (true binary labels).
+    - y_scores: Predicted scores or probabilities for positive class.
+    - plot: Boolean to define if computed ROC AUC shall be plotted or not. Default is False.
+    - title_suffix: string to be appended to the plot title in case plot=True.
+    If plot is False, this parameter is ignored.
+
+    Returns:
+    - fpr: False Positive Rate (1 - Specificity).
+    - tpr: True Positive Rate (Sensitivity).
+    - roc_auc: Area Under the ROC Curve (AUC).
+    """
+    fpr, tpr, thresholds = roc_curve(y_true.flatten().detach().numpy(), y_scores.flatten().detach().numpy())
+    best_index = np.argmax(tpr - fpr)
+    print("Youden Index: ", thresholds[best_index])
+    print(f"FPR: {fpr[best_index]}, TPR: {tpr[best_index]}")
+    print(f"Specificity: {1 - fpr[best_index]}, Sensitivity: {tpr[best_index]}")
+    roc_auc = auc(fpr, tpr)
+
+    if plot:
+        # ROC plot
+        if title_suffix is None:
+            title_suffix = ''
+        plt.title('Receiver Operating Characteristic: ' + title_suffix)
+        plt.plot(fpr, tpr, 'b', label='AUC = %0.2f' % roc_auc)
+        plt.legend(loc='lower right')
+        plt.plot([0, 1], [0, 1], 'r--')
+        plt.xlim([0, 1])
+        plt.ylim([0, 1])
+        plt.ylabel('True Positive Rate')
+        plt.xlabel('False Positive Rate')
+        #plt.show()
+
+        title_suffix = title_suffix.replace(' ', '_')
+        title_suffix = title_suffix.replace('resnet50_', '')
+        title_suffix = title_suffix.replace('resnet18_', '')
+        roc_key = "roc_auc/" + title_suffix
+        spec_key = "Specificity (Youden)/" + title_suffix
+        sens_key = "Sensitivity (Youden)/" + title_suffix
+        wandb.log({
+            roc_key: wandb.Image(plt), 
+            spec_key: 1 - fpr[best_index], 
+            sens_key: tpr[best_index]
+        })
+        plt.close()
+
+        # TPR against 1-FPR: helps to find optimal threshold
+        i = np.arange(len(tpr))  # index for df
+        roc = pd.DataFrame(
+            {'fpr': pd.Series(fpr, index=i), 'tpr': pd.Series(tpr, index=i), '1-fpr': pd.Series(1 - fpr, index=i),
+             'tf': pd.Series(tpr - (1 - fpr), index=i), 'thresholds': pd.Series(thresholds, index=i)})
+        roc.iloc[(roc.tf - 0).abs().argsort()[:1]]
+        # Plot tpr vs 1-fpr
+        fig, ax = pl.subplots()
+        pl.plot(roc['tpr'])
+        pl.plot(roc['1-fpr'], color='red')
+        pl.xlabel('1-False Positive Rate')
+        pl.ylabel('True Positive Rate')
+        pl.title('Receiver operating characteristic')
+        ax.set_xticklabels([])
+        wandb.log({
+            "Thresholds/" + title_suffix: fig
+        })
+        pl.close(fig)
+        roc_values = {"roc": roc, "fpr": fpr, "tpr": tpr, "thresholds": thresholds, "best_index_val": best_index, "roc_auc_val": roc_auc}
+    else:
+        roc_values = {"fpr": fpr, "tpr": tpr, "thresholds": thresholds, "best_index_val": best_index, "roc_auc_val": roc_auc}
+    return roc_values
+
+
+def auc_bootstrapping(y_true, y_score, bootstrapping=1000, drop_intermediate=False):
+    tprs, aucs, thrs, sens, spec = [], [], [], [], []
+    mean_fpr = np.linspace(0, 1, 100)  # fixed set for analysation later
+    np.random.seed(0)
+    rand_idxs = np.random.randint(0, len(y_true), size=(bootstrapping, len(y_true))) # Note: with replacement
+    for rand_idx in rand_idxs:
+        y_true_set = y_true[rand_idx]
+        y_score_set = y_score[rand_idx]
+
+        # calc aucs
+        fpr, tpr, thresholds = roc_curve(y_true_set, y_score_set, drop_intermediate=drop_intermediate)
+        tpr_interp = np.interp(mean_fpr, fpr, tpr) # must be interpolated to gain constant/equal fpr positions
+        tprs.append(tpr_interp)
+        aucs.append(auc(fpr, tpr))
+        optimal_idx = np.argmax(tpr - fpr)
+        thrs.append(thresholds[optimal_idx])
+        # sensitivity and specificity
+        sens.append(tpr[optimal_idx])
+        spec.append(1 - fpr[optimal_idx])
+    return tprs, aucs, thrs, mean_fpr, sens, spec
+
+def plot_aucs(mean_fpr, tprs1, tprs2, aucs1, aucs2, model_name, file_name):
+    # Durchschnittliche TPR und Standardabweichung für Modell 1
+    mean_tpr_1 = np.mean(tprs1 , axis=0)
+    std_tpr_1  = np.std(tprs1 , axis=0)
+
+     # Durchschnittliche TPR und Standardabweichung für Modell 2
+    mean_tpr_2 = np.mean(tprs2 , axis=0)
+    std_tpr_2  = np.std(tprs2 , axis=0)
+
+    # Plotte die ROC-Kurven
+    plt.figure()
+    plt.plot(mean_fpr, mean_tpr_1, color='blue',label='FDP (AUC={:.3f})'.format(np.mean(aucs1)))
+
+    # diagonal
+    plt.plot([0, 1], [0, 1], 'r--')
+    plt.xlim([0, 1])
+    plt.ylim([0, 1])
+    plt.fill_between(mean_fpr ,
+                    mean_tpr_1 - std_tpr_1,
+                    mean_tpr_1 + std_tpr_1,
+                    color='blue', alpha=0.3)
+
+    plt.plot(mean_fpr, mean_tpr_2, color='orange',label='AP (AUC={:.3f})'.format(np.mean(aucs2)))
+    plt.fill_between(mean_fpr,
+                    mean_tpr_2 - std_tpr_2,
+                    mean_tpr_2 + std_tpr_2,
+                    color='orange', alpha=0.3)
+
+    plt.xlabel('False Positive Rate')
+    plt.ylabel('True Positive Rate')
+    plt.title(f'Receiver Operating Characteristic for {model_name}')
+    plt.legend(loc='lower right')
+    plt.savefig(file_name)
+
+
+if __name__ == "__main__":
+    """
+    Bootstrap AUC/sensitivity/specificity of the abbreviated against the full protocol from the
+    per-fold result pickles written by the comparison scripts.
+
+        python -m auto_detect_breast_mri.evaluation.metrics --model-name resnet50 [--result-root <folder>]
+    """
+    import argparse
+
+    from auto_detect_breast_mri.config import get_config, resolve_path
+
+    parser = argparse.ArgumentParser(description="Bootstrap the abrv/full comparison of one architecture.")
+    parser.add_argument("--model-name", default="resnet50", help="architecture prefix of the pickles")
+    parser.add_argument("--result-root", default=None,
+                        help="folder holding the *_roc-dict.pkl files. Config key: output_root")
+    parser.add_argument("--pickle-pattern", default="{}_{}_split{}-0_roc-dict.pkl",
+                        help="file name pattern with model name, protocol and fold placeholders")
+    parser.add_argument("--folds", type=int, nargs="+", default=[0, 1, 2, 3, 4])
+    args = parser.parse_args()
+
+    result_root = resolve_path(args.result_root, "output_root", "folder with the result pickles")
+    pickle_file_name = os.path.join(result_root, args.pickle_pattern)
+    model_name = args.model_name
+    wandb.init(**get_config().wandb_init_kwargs(name=model_name))
+
+    full_predictions = []
+    full_probs = []
+    abrv_predictions = []
+    abrv_probs = []
+    ground_truth = []
+    for fold in args.folds:
+        for prot in ["abrv", "full"]:
+            curr_pickle_file = pickle_file_name.format(model_name, prot, fold)
+            with open(curr_pickle_file, 'rb') as pickle_file:
+                loaded_data = pickle.load(pickle_file)
+                if prot == "full":
+                    ground_truth.append(loaded_data["y_true"])
+                    full_predictions.append(loaded_data["y_pred"])
+                    full_probs.append(loaded_data["y_probs"])
+                else:
+                    abrv_predictions.append(loaded_data["y_pred"])
+                    abrv_probs.append(loaded_data["y_probs"])
+    print("model " + model_name + "full vs abrv: ")
+
+    # compute the evaluation metrices:
+    ### bootstrapping for AUC, Sensitivities and Specificities
+    tprs_full, aucs_full, thrs_full, mean_fpr_full, sens_full, spec_full = auc_bootstrapping(np.concatenate(ground_truth).flatten(), np.concatenate(full_probs).flatten())
+    tprs_abrv, aucs_abrv, thrs_abrv, mean_fpr_abrv, sens_abrv, spec_abrv = auc_bootstrapping(np.concatenate(ground_truth).flatten(), np.concatenate(abrv_probs).flatten())
+    
+    p_value = delong_roc_test(np.concatenate(ground_truth), np.concatenate(abrv_predictions), np.concatenate(full_predictions))
+    print("Total p value: " + str(p_value))
+
+    # plot roc curves of abrv and full in one figure:
+    plot_filename = pickle_file_name.split("_split")[0].format(model_name, "bootstrap.png")
+    plot_aucs(mean_fpr_full, tprs_full, tprs_abrv, aucs_full, aucs_abrv, model_name, plot_filename)
+    wandb.log({"p_value": p_value, 
+                "AUC-mean/full": np.mean(aucs_full),
+                "AUC-mean/abrv": np.mean(aucs_abrv), 
+                "AUC-std/full": np.std(aucs_full),
+                "AUC-std/abrv": np.std(aucs_abrv), 
+                "Sensitivity-mean/full": np.mean(sens_full),
+                "Sensitivity-mean/abrv": np.mean(sens_abrv), 
+                "Sensitivity-std/full": np.std(sens_full),
+                "Sensitivity-std/abrv": np.std(sens_abrv), 
+                "Specificity-mean/full": np.mean(spec_full),
+                "Specificity-mean/abrv": np.mean(spec_abrv), 
+                "Specificity-std/full": np.std(spec_full),
+                "Specificity-std/abrv": np.std(spec_abrv), 
+                })
+    
+    bootstrap_result = {"tprs_full": tprs_full, "aucs_full": aucs_full, "thrs_full": thrs_full, "mean_fpr_full": mean_fpr_full,
+                        "tprs_abrv": tprs_abrv, "aucs_abrv": aucs_abrv, "thrs_abrv": thrs_abrv, "mean_fpr_abrv": mean_fpr_abrv,
+                        "p_value": p_value, "resulting_plot": plot_filename}
+    bootstrap_pickle_name = plot_filename.replace(".png", ".pkl")
+    with open(bootstrap_pickle_name, 'wb') as pickle_file:
+        pickle.dump(bootstrap_result, pickle_file)
+    

auto_detect_breast_mri/evaluation/oof_table.py

@@ -0,0 +1,327 @@
+"""
+Assembly and integrity checking of the out-of-fold prediction table.
+
+The table has one row per breast, with the paired AP and FDP score of every architecture that was
+predicted side by side, so that every downstream analysis works on exactly the same cases:
+
+    patient_id, outer_fold, examination_id, side, label,
+    AP_score_ResNet18, FDP_score_ResNet18, AP_score_ResNet50, FDP_score_ResNet50
+
+Which score columns exist follows from the prediction files that are actually there: predicting only
+one architecture yields a table with that architecture's two columns. Both protocols of an
+architecture are required, since the whole analysis is the paired AP vs FDP comparison.
+"""
+
+from auto_detect_breast_mri.config import column, indication_labels, normalise_indication_code
+
+import glob
+import os
+import re
+
+import pandas as pd
+
+ARCHITECTURE_LABELS = {'resnet18': 'ResNet18', 'resnet50': 'ResNet50'}
+PROTOCOL_LABELS = {'abrv': 'AP', 'full': 'FDP'}
+KEY_COLUMNS = ['examination_id', 'side']
+SIDES = ('left', 'right')
+# oof_<architecture>_fold<k>[_frac=<f>].csv, as written by predict_oof.py
+FILE_PATTERN = re.compile(r'^oof_(?P<architecture>[A-Za-z0-9]+)_fold(?P<fold>\d+)'
+                          r'(?:_frac=(?P<fraction>[0-9.]+))?\.csv$')
+UNKNOWN_INDICATION = 'unknown'
+CASE_COLUMNS = ['patient_id', 'outer_fold', 'examination_id', 'side', 'label']
+
+
+def score_columns(architecture):
+    """The AP and the FDP score column of one architecture label, e.g. 'ResNet18'."""
+    return [f'{protocol}_score_{architecture}' for protocol in PROTOCOL_LABELS.values()]
+
+
+def score_columns_of(table):
+    """
+    The score columns present in a table, grouped by architecture and in a stable order.
+    :return: list of column names, AP before FDP within each architecture
+    """
+    return [name for architecture in ARCHITECTURE_LABELS.values()
+            for name in score_columns(architecture) if name in table.columns]
+
+
+def architectures_of(table):
+    """The architecture labels a table carries both scores of."""
+    return [architecture for architecture in ARCHITECTURE_LABELS.values()
+            if all(name in table.columns for name in score_columns(architecture))]
+
+
+def read_metadata(metadata_file, wanted_ids=None):
+    """
+    Read the metadata export and give it an `examination_id` column that matches the split CSVs.
+
+    The metadata export appends a two digit running number to the examination ID ('9132094301' for
+    '91320943'), which the split files do not carry. Matching the raw IDs therefore resolves nothing
+    at all on the UKA export. The ID is taken raw when that matches, and trimmed by the two trailing
+    digits otherwise, the same fallback the dataset loader and dataset_stats apply.
+
+    :param metadata_file: str, CSV/XLSX metadata export
+    :param wanted_ids: iterable of examination IDs to match against, or None to always trim
+    :return: (metadata frame with an added 'examination_id' column, description of how it matched)
+    """
+    reader = pd.read_excel if str(metadata_file).endswith(('.xlsx', '.xls')) else pd.read_csv
+    metadata = reader(metadata_file, dtype=str)
+    id_column = column('metadata_examination_id')
+    if id_column not in metadata.columns:
+        raise ValueError(f"{metadata_file} has no column '{id_column}'.")
+
+    raw = metadata[id_column].astype(str).str.strip()
+    trimmed = raw.str[:-2]
+    if wanted_ids is None:
+        metadata['examination_id'], how = trimmed, "trimmed by the two trailing digits"
+    else:
+        wanted = set(map(str, wanted_ids))
+        matched_raw, matched_trimmed = raw.isin(wanted).sum(), trimmed.isin(wanted).sum()
+        if matched_trimmed > matched_raw:
+            metadata['examination_id'] = trimmed
+            how = (f"trimmed by the two trailing digits ({matched_trimmed} matches, "
+                   f"{matched_raw} without trimming)")
+        else:
+            metadata['examination_id'] = raw
+            how = f"used as they are ({matched_raw} matches)"
+    return metadata, how
+
+
+def _unique_mapping(metadata, value_column, what):
+    """
+    dict examination_id -> value, dropping IDs that carry conflicting values.
+    :return: (mapping, list of the dropped ambiguous IDs)
+    """
+    pairs = metadata[['examination_id', value_column]].dropna().drop_duplicates()
+    counts = pairs.groupby('examination_id').size()
+    ambiguous = sorted(counts[counts > 1].index)
+    pairs = pairs[~pairs['examination_id'].isin(ambiguous)]
+    if ambiguous and what == 'women':
+        raise ValueError(f"{len(ambiguous)} examinations map to more than one woman, "
+                         f"e.g. {ambiguous[:5]}. The clustering would be wrong.")
+    return dict(zip(pairs['examination_id'], pairs[value_column])), ambiguous
+
+
+def load_patient_map(metadata_file, wanted_ids=None):
+    """
+    Map examination IDs to the woman they belong to.
+    :param metadata_file: str
+        CSV holding the examination ID and the patient ID column.
+    :param wanted_ids: the examination IDs to match against, see read_metadata
+    :return: dict examination_id -> patient_id
+    """
+    metadata, how = read_metadata(metadata_file, wanted_ids)
+    patient_column = column('patient_id')
+    if patient_column not in metadata.columns:
+        raise ValueError(f"{metadata_file} has no column '{patient_column}'; cannot resolve women.")
+    print(f"  patient IDs from {metadata_file}: examination IDs {how}")
+    mapping, _ = _unique_mapping(metadata, patient_column, 'women')
+    return mapping
+
+
+def load_indication_map(metadata_file, wanted_ids=None):
+    """
+    Map (examination ID, side) to the indication code of that breast.
+
+    The indication is recorded per side in the wide metadata export, so it lands on exactly one
+    breast, i.e. one row of the out-of-fold table. Codes are normalised to strings ('1.0' -> '1').
+
+    :return: (dict (examination_id, side) -> code, report dict)
+    """
+    metadata, how = read_metadata(metadata_file, wanted_ids)
+    mapping, report = {}, {'id_matching': how, 'ambiguous': {}, 'per_side': {}}
+    for side in SIDES:
+        indication_column = column(f'indication_{side}')
+        if indication_column not in metadata.columns:
+            raise ValueError(f"{metadata_file} has no column '{indication_column}'; "
+                             f"cannot build the indication subgroups.")
+        per_side, ambiguous = _unique_mapping(metadata, indication_column, f'indication {side}')
+        report['ambiguous'][side] = ambiguous
+        report['per_side'][side] = len(per_side)
+        for examination_id, code in per_side.items():
+            mapping[(examination_id, side)] = normalise_indication_code(code)
+    return mapping, report
+
+
+def attach_indication(table, metadata_file):
+    """
+    Add an `indication` column to an out-of-fold table, one code per breast.
+
+    Breasts whose examination is missing from the metadata, or whose indication field is empty, get
+    UNKNOWN_INDICATION rather than being dropped: whether they are analysed as their own subgroup or
+    excluded is the caller's decision, not this function's.
+
+    :return: (table with the added column, report dict describing the match)
+    """
+    mapping, report = load_indication_map(metadata_file, table['examination_id'].unique())
+    table = table.copy()
+    keys = list(zip(table['examination_id'].astype(str), table['side'].astype(str).str.lower()))
+    table['indication'] = [mapping.get(key, UNKNOWN_INDICATION) for key in keys]
+
+    labels = indication_labels()
+    table['indication_label'] = [labels.get(code, f"code {code}") if code != UNKNOWN_INDICATION
+                                 else UNKNOWN_INDICATION for code in table['indication']]
+    report['breasts'] = len(table)
+    report['without_indication'] = int((table['indication'] == UNKNOWN_INDICATION).sum())
+    report['per_code'] = table['indication'].value_counts().to_dict()
+    return table, report
+
+
+def load_oof_table(prediction_dir, metadata_file):
+    """
+    Read the per fold prediction files written by predict_oof.py and pivot them into the wide table.
+    Only breasts for which every score of every usable architecture exists are kept; the dropped ones
+    are reported separately. An architecture counts as usable once both its abrv and its full
+    prediction files are present.
+    :param prediction_dir: str
+        folder holding oof_<architecture>_fold<k>.csv
+    :param metadata_file: str
+        CSV mapping AnforderungsNrE to PNR
+    :return: (table, report) with report describing what was dropped while merging
+    """
+    files = sorted(glob.glob(os.path.join(prediction_dir, 'oof_*_fold*.csv')))
+    if not files:
+        raise FileNotFoundError(f"No oof_*_fold*.csv found in {prediction_dir}. Run predict_oof.py first.")
+
+    frames, folds_of_architecture, fractions = [], {}, {}
+    for path in files:
+        parsed = FILE_PATTERN.match(os.path.basename(path))
+        if parsed is None or parsed.group('architecture') not in ARCHITECTURE_LABELS:
+            raise ValueError(f"Cannot tell the architecture and fold from file name {path}.")
+        architecture = ARCHITECTURE_LABELS[parsed.group('architecture')]
+        frame = pd.read_csv(path, dtype={'examination_id': str, 'side': str})
+        frame['architecture'] = architecture
+        frames.append(frame)
+        folds_of_architecture.setdefault(architecture, set()).add(int(parsed.group('fold')))
+        fractions.setdefault(parsed.group('fraction') or 'full', []).append(os.path.basename(path))
+    long_table = pd.concat(frames, ignore_index=True)
+
+    # Predictions of different training fractions are different models. Averaging or pivoting them
+    # into one table would silently mix them, so refuse rather than produce a meaningless table.
+    if len(fractions) > 1:
+        listing = "; ".join(f"{name}: {len(paths)} files" for name, paths in sorted(fractions.items()))
+        raise ValueError(f"{prediction_dir} mixes predictions of several training fractions "
+                         f"({listing}). Keep one fraction per folder.")
+
+    # A fold that only some architectures predicted is dropped completely further down, because a
+    # breast needs every score column. Say so here instead of letting it vanish into a count.
+    all_folds = set().union(*folds_of_architecture.values())
+    for architecture, folds in sorted(folds_of_architecture.items()):
+        if folds != all_folds:
+            print(f"  WARNING: {architecture} has no prediction file for fold(s) "
+                  f"{sorted(all_folds - folds)}; every breast of those folds will be dropped "
+                  f"because it cannot have a complete set of scores.")
+    long_table['column'] = (long_table['protocol'].map(PROTOCOL_LABELS) + '_score_'
+                            + long_table['architecture'])
+
+    duplicates = long_table.duplicated(subset=KEY_COLUMNS + ['column'], keep=False)
+    if duplicates.any():
+        offending = long_table.loc[duplicates, KEY_COLUMNS + ['column', 'outer_fold']]
+        raise ValueError(f"{duplicates.sum()} duplicated predictions for the same breast and model, "
+                         f"e.g.\n{offending.head().to_string()}")
+
+    wide = long_table.pivot(index=KEY_COLUMNS, columns='column', values='score').reset_index()
+    wide.columns.name = None
+
+    # fold and label are properties of the case, not of the model: take them from the long table
+    case_info = (long_table[KEY_COLUMNS + ['outer_fold', 'label']]
+                 .drop_duplicates(subset=KEY_COLUMNS))
+    inconsistent = (long_table.groupby(KEY_COLUMNS)[['outer_fold', 'label']].nunique() > 1)
+    inconsistent = inconsistent[inconsistent.any(axis=1)]
+    if len(inconsistent):
+        raise ValueError(f"{len(inconsistent)} breasts carry conflicting fold or label across the "
+                         f"prediction files, e.g. {list(inconsistent.index[:5])}.")
+    table = case_info.merge(wide, on=KEY_COLUMNS, how='outer')
+
+    patient_map = load_patient_map(metadata_file, table['examination_id'].unique())
+    table['patient_id'] = table['examination_id'].map(patient_map)
+
+    report = {'cases_read': len(table)}
+    unmapped = table['patient_id'].isna()
+    report['dropped_without_patient_id'] = int(unmapped.sum())
+    report['dropped_without_patient_id_examples'] = sorted(table.loc[unmapped, 'examination_id'].unique()[:10])
+    table = table[~unmapped]
+
+    # an architecture is only usable if both of its protocols were predicted: the analysis is the
+    # paired AP vs FDP comparison, a lone AP or FDP column carries no comparison
+    architectures = architectures_of(table)
+    half_predicted = sorted(set(score_columns_of(table))
+                            - {name for architecture in architectures for name in score_columns(architecture)})
+    if not architectures:
+        raise ValueError(f"No architecture in {prediction_dir} has both an abrv and a full prediction "
+                         f"file; found only {half_predicted or 'no score columns'}.")
+    report['architectures'] = architectures
+    report['ignored_columns_without_counterpart'] = half_predicted
+    columns = [name for architecture in architectures for name in score_columns(architecture)]
+
+    incomplete = table[columns].isna().any(axis=1)
+    report['dropped_incomplete_scores'] = int(incomplete.sum())
+    report['incomplete_per_column'] = {name: int(table.loc[incomplete, name].isna().sum())
+                                       for name in columns}
+    table = table[~incomplete]
+
+    table = table[CASE_COLUMNS + columns]
+    table = table.sort_values(['outer_fold', 'patient_id', 'examination_id', 'side']).reset_index(drop=True)
+    report['cases_kept'] = len(table)
+    return table, report
+
+
+def check_integrity(table):
+    """
+    The data integrity checks required before any inference is run.
+    :param table: the wide out-of-fold table
+    :return: (checks, summary) where checks maps a check name to (passed, message)
+    """
+    checks = {}
+
+    columns = score_columns_of(table)
+    complete = table[columns].notna().all(axis=1).all()
+    checks['identical_cases_per_AP_FDP_pair'] = (
+        bool(complete),
+        f"every kept breast has an AP and an FDP score for {', '.join(architectures_of(table))}"
+        if complete else f"some breasts lack one of the {len(columns)} scores")
+
+    duplicated = table.duplicated(subset=KEY_COLUMNS).sum()
+    checks['no_duplicate_breasts'] = (
+        duplicated == 0,
+        "no breast appears twice" if duplicated == 0 else f"{duplicated} duplicated breasts")
+
+    folds_per_patient = table.groupby('patient_id')['outer_fold'].nunique()
+    straddling = folds_per_patient[folds_per_patient > 1]
+    checks['no_patient_in_several_test_folds'] = (
+        len(straddling) == 0,
+        f"all {len(folds_per_patient)} women sit in exactly one test fold"
+        if len(straddling) == 0 else
+        f"{len(straddling)} women appear in several test folds, e.g. {list(straddling.index[:5])}")
+
+    exams_per_patient = table.groupby('patient_id')['examination_id'].nunique()
+    repeated = exams_per_patient[exams_per_patient > 1]
+    folds_of_repeated = table[table['patient_id'].isin(repeated.index)].groupby('patient_id')['outer_fold'].nunique()
+    clustered = bool((folds_of_repeated == 1).all()) if len(repeated) else True
+    checks['repeated_examinations_clustered'] = (
+        clustered,
+        f"{len(repeated)} women have several examinations, all within one fold"
+        if clustered else "repeated examinations of a woman are spread over folds")
+
+    labels_ok = set(table['label'].unique()) <= {0, 1}
+    checks['labels_binary'] = (
+        labels_ok, "labels are binary" if labels_ok else f"unexpected labels {sorted(set(table['label']))}")
+
+    per_fold_classes = table.groupby('outer_fold')['label'].agg(['sum', 'count'])
+    both_classes = bool(((per_fold_classes['sum'] > 0) & (per_fold_classes['sum'] < per_fold_classes['count'])).all())
+    checks['both_classes_in_every_fold'] = (
+        both_classes, "every fold holds malignant and benign breasts"
+        if both_classes else "a fold holds only one class, its AUC is undefined")
+
+    summary = {
+        'breasts': len(table),
+        'examinations': table['examination_id'].nunique(),
+        'women': table['patient_id'].nunique(),
+        'malignant_breasts': int(table['label'].sum()),
+        'women_with_repeated_examinations': int(len(repeated)),
+        'max_examinations_per_woman': int(exams_per_patient.max()) if len(exams_per_patient) else 0,
+        'per_fold': table.groupby('outer_fold').agg(
+            breasts=('label', 'size'), malignant=('label', 'sum'),
+            women=('patient_id', 'nunique')).to_dict('index'),
+    }
+    return checks, summary

auto_detect_breast_mri/evaluation/subgroups.py

@@ -0,0 +1,186 @@
+"""
+Performance within a subgroup of the out-of-fold table, e.g. per indication.
+
+This is an exploratory companion to the primary AP vs FDP analysis, not a second confirmatory one.
+Each subgroup is analysed with exactly the machinery of the primary analysis, only restricted to its
+own rows: AUCs are computed *within* an outer fold and combined with weights that are fixed before
+resampling, and the bootstrap redraws whole women so that both breasts and all repeated examinations
+of one woman move together.
+
+Three things are worth keeping in mind when reading the output:
+
+* The fold weights are recomputed inside the subgroup. A subgroup's weights are therefore not the
+  weights of the primary analysis, and its delta is not a decomposition of the overall delta.
+* Subgroups are small. A fold that holds only benign breasts has no AUC, is dropped, and the
+  remaining folds are renormalized; `folds_used` reports how many actually carried information.
+* Nothing here is multiplicity adjusted across subgroups, and no non-inferiority verdict is derived.
+  A wide interval in a small subgroup means "not measurable here", never "inferior".
+"""
+
+import numpy as np
+
+from auto_detect_breast_mri.evaluation.cluster_bootstrap import (FoldClusterBootstrap, auc,
+                                                                 percentile_interval,
+                                                                 weighted_fold_difference)
+from auto_detect_breast_mri.evaluation.oof_table import score_columns
+
+# A subgroup below this many breasts, or with fewer than this many of either class, is reported with
+# its counts but without a bootstrap: the interval would be an artefact of a handful of cases.
+MIN_BREASTS = 20
+MIN_PER_CLASS = 5
+
+
+def weighted_fold_auc(rows_of_fold, labels, scores, weights):
+    """
+    One AUC per fold, combined with the fixed weights, folds without both classes dropped.
+    :return: (combined AUC, dict fold -> AUC, number of folds that contributed)
+    """
+    per_fold, combined, total_weight = {}, 0.0, 0.0
+    for fold, rows in rows_of_fold.items():
+        value = auc(labels[rows], scores[rows])
+        per_fold[fold] = value
+        if not np.isnan(value):
+            combined += weights[fold] * value
+            total_weight += weights[fold]
+    if total_weight == 0:
+        return np.nan, per_fold, 0
+    return combined / total_weight, per_fold, int(sum(not np.isnan(v) for v in per_fold.values()))
+
+
+def describe_subgroup(rows, table):
+    """Counts that decide whether a subgroup can carry an analysis at all."""
+    part = table.iloc[rows]
+    labels = part['label'].to_numpy()
+    return {'breasts': len(part),
+            'women': int(part['patient_id'].nunique()),
+            'examinations': int(part['examination_id'].nunique()),
+            'malignant': int((labels == 1).sum()),
+            'benign': int((labels == 0).sum()),
+            'folds_with_both_classes': int(sum(
+                1 for _, fold_part in part.groupby('outer_fold')
+                if 0 < fold_part['label'].sum() < len(fold_part)))}
+
+
+def analysable(counts, min_breasts=MIN_BREASTS, min_per_class=MIN_PER_CLASS):
+    """
+    Whether a subgroup is worth bootstrapping at all.
+    :return: (bool, reason) with reason empty when it is analysable
+    """
+    if counts['breasts'] < min_breasts:
+        return False, f"only {counts['breasts']} breasts (minimum {min_breasts})"
+    if counts['malignant'] < min_per_class or counts['benign'] < min_per_class:
+        return False, (f"only {counts['malignant']} malignant / {counts['benign']} benign "
+                       f"(minimum {min_per_class} each)")
+    if counts['folds_with_both_classes'] == 0:
+        return False, "no outer fold holds both classes, every fold level AUC is undefined"
+    return True, ""
+
+
+def analyse_subgroup(table, rows, architectures, replications, weighting, seed, stratify):
+    """
+    AUC of every model and the paired AP minus FDP difference inside one subgroup.
+
+    :param table: the wide out-of-fold table
+    :param rows: positional row indices of the subgroup within `table`
+    :param architectures: architecture labels to analyse
+    :return: dict with the counts, the per architecture results and, if the subgroup is too small,
+        a 'skipped' reason instead of the bootstrap results
+    """
+    counts = describe_subgroup(rows, table)
+    usable, reason = analysable(counts)
+    result = {'counts': counts, 'architectures': {}}
+    if not usable:
+        result['skipped'] = reason
+        return result
+
+    part = table.iloc[rows].reset_index(drop=True)
+    labels = part['label'].to_numpy()
+    bootstrap = FoldClusterBootstrap(part['outer_fold'].to_numpy(), part['patient_id'].to_numpy(),
+                                     labels, stratify_by_outcome=stratify)
+    weights = bootstrap.fold_weights(weighting)
+    observed_rows = bootstrap.observed_rows()
+    scores = {architecture: tuple(part[name].to_numpy() for name in score_columns(architecture))
+              for architecture in architectures}
+
+    for architecture in architectures:
+        ap_scores, fdp_scores = scores[architecture]
+        auc_ap, per_fold_ap, folds_ap = weighted_fold_auc(observed_rows, labels, ap_scores, weights)
+        auc_fdp, per_fold_fdp, _ = weighted_fold_auc(observed_rows, labels, fdp_scores, weights)
+        delta, _ = weighted_fold_difference(observed_rows, labels, ap_scores, fdp_scores, weights)
+        result['architectures'][architecture] = {
+            'auc_ap': auc_ap, 'auc_fdp': auc_fdp, 'observed_delta': delta,
+            'folds_used': folds_ap,
+            'per_fold': {fold: (per_fold_ap[fold], per_fold_fdp[fold]) for fold in per_fold_ap},
+            'replications': np.full(replications, np.nan)}
+
+    # one rng for the whole subgroup, so all architectures see the same resampled women
+    rng = np.random.default_rng(seed)
+    degenerate = 0
+    for replication in range(replications):
+        rows_of_fold = bootstrap.resample_rows(rng)
+        for architecture in architectures:
+            ap_scores, fdp_scores = scores[architecture]
+            delta, _ = weighted_fold_difference(rows_of_fold, labels, ap_scores, fdp_scores, weights)
+            result['architectures'][architecture]['replications'][replication] = delta
+            degenerate += int(np.isnan(delta))
+
+    for architecture in architectures:
+        entry = result['architectures'][architecture]
+        entry['ci_95'] = percentile_interval(entry['replications'], 0.95)
+        entry['bootstrap_se'] = float(np.nanstd(entry['replications'], ddof=1))
+        entry['degenerate_replications'] = int(np.isnan(entry['replications']).sum())
+    result['weights'] = weights
+    result['degenerate_replications'] = degenerate
+    return result
+
+
+def analyse_by_group(table, group_column, architectures, replications=2000, weighting='pairs',
+                     seed=0, stratify=True, label_column=None):
+    """
+    Run analyse_subgroup for every value of `group_column`, plus one 'all' row over the whole table.
+
+    :param table: the wide out-of-fold table
+    :param group_column: column holding the subgroup key, e.g. 'indication'
+    :param label_column: optional column holding a readable name of the subgroup
+    :return: dict subgroup key -> result of analyse_subgroup, 'all' first
+    """
+    results = {}
+    everything = np.arange(len(table))
+    results['all'] = analyse_subgroup(table, everything, architectures, replications, weighting,
+                                      seed, stratify)
+    results['all']['label'] = 'all breasts'
+
+    positions = {key: np.flatnonzero((table[group_column] == key).to_numpy())
+                 for key in table[group_column].unique()}
+    for key in sorted(positions, key=lambda k: (-len(positions[k]), str(k))):
+        print(f"  subgroup {key}: {len(positions[key])} breasts", flush=True)
+        entry = analyse_subgroup(table, positions[key], architectures, replications, weighting,
+                                 seed, stratify)
+        entry['label'] = (table.iloc[positions[key]][label_column].iloc[0]
+                          if label_column else str(key))
+        results[key] = entry
+    return results
+
+
+def to_frame(results, architectures):
+    """
+    The per subgroup results as a flat table, one row per (subgroup, architecture).
+    :return: list of dicts, ready for pandas
+    """
+    rows = []
+    for key, entry in results.items():
+        counts = entry['counts']
+        for architecture in architectures:
+            row = {'subgroup': key, 'label': entry.get('label', str(key)),
+                   'architecture': architecture, **counts}
+            values = entry['architectures'].get(architecture)
+            if values is None:
+                row['skipped'] = entry.get('skipped', '')
+            else:
+                row.update({'auc_AP': values['auc_ap'], 'auc_FDP': values['auc_fdp'],
+                            'delta_AP_minus_FDP': values['observed_delta'],
+                            'ci95_low': values['ci_95'][0], 'ci95_high': values['ci_95'][1],
+                            'bootstrap_se': values['bootstrap_se'],
+                            'folds_used': values['folds_used'], 'skipped': ''})
+            rows.append(row)
+    return rows

auto_detect_breast_mri/models/__init__.py

@@ -0,0 +1 @@
+"""Network definitions and checkpoint IO."""

auto_detect_breast_mri/models/checkpoints.py

@@ -0,0 +1,53 @@
+"""Saving and loading model checkpoints."""
+
+import torch
+from torch import nn
+
+
+def save_model(model: nn.Module, parent_dir: str, filename: str):
+    if all([model, parent_dir, filename]):
+        torch.save(model.state_dict(), parent_dir + '/' + filename + '.pth')
+
+
+def load_pretrained_model(model, model_path, DEVICE):
+    '''
+    Load a model from '.pth' file to use the pretrained model in our later training loop or only evaluate it.
+    '''
+    checkpoint = torch.load(model_path, torch.device(DEVICE), weights_only=True)
+    state_dict = {}
+    if 'state_dict' in checkpoint.keys():
+        for k, v in checkpoint['state_dict'].items():
+            if k.startswith('module.'):
+                key = k.replace('module.', '')
+            else:
+                key = k
+            # check dimension of checkpoint value:
+            desired_shape = model.state_dict()[key].shape
+            if desired_shape != v.shape:
+                pretrained_layer_tensor = torch.empty(desired_shape)
+                (l, m, n, o, p) = desired_shape
+                (a, b, c, d, e) = v.shape  # given pretrained parameters shape
+                pretrained_layer_tensor[:a, :b, :c, :d, :e] = v
+
+                # Duplicate values along the additional dimensions
+                if a < l:
+                    for i in range(l):
+                        pretrained_layer_tensor[a + i:a + i + 1, :, :, :, :] = pretrained_layer_tensor[:a, :, :, :, :]
+                if b < m:
+                    for j in range(m):
+                        pretrained_layer_tensor[:, b + j:b + j + 1, :, :, :] = pretrained_layer_tensor[:, :b, :, :, :]
+                if c < n:
+                    for k in range(n):
+                        pretrained_layer_tensor[:, :, c + k:c + k + 1, :, :] = pretrained_layer_tensor[:, :, :c, :, :]
+                if d < o:
+                    for l in range(o):
+                        pretrained_layer_tensor[:, :, :, d + l:d + l + 1, :] = pretrained_layer_tensor[:, :, :, :d, :]
+                if e < p:
+                    for m in range(p):
+                        pretrained_layer_tensor[:, :, :, :, e + m:e + m + 1] = pretrained_layer_tensor[:, :, :, :, :e]
+        for k, v in model.state_dict().items():
+            if k not in state_dict.keys():
+                state_dict[k] = model.state_dict()[k]
+        return state_dict
+    else:
+        return checkpoint

auto_detect_breast_mri/models/resnets.py

@@ -0,0 +1,30 @@
+from monai.networks import nets
+
+LEN_SINGLE = 1
+LEN_3 = 3
+LEN_ABRV = 2
+LEN_FULL = 6
+
+resnet18_single = nets.ResNet('basic', [2, 2, 2, 2], [64, 128, 256, 512],
+                              n_input_channels=LEN_SINGLE, num_classes=1)
+resnet50_single = nets.ResNet('bottleneck', [3, 4, 6, 3], [64, 128, 256, 512],
+                              n_input_channels=LEN_SINGLE, num_classes=1)
+resnet18_abrv_t2 = nets.ResNet('basic', [2, 2, 2, 2], [64, 128, 256, 512],
+                            n_input_channels=LEN_3, num_classes=1)
+resnet18_abrv = nets.ResNet('basic', [2, 2, 2, 2], [64, 128, 256, 512],
+                            n_input_channels=LEN_ABRV, num_classes=1)
+resnet50_abrv = nets.ResNet('bottleneck', [3, 4, 6, 3], [64, 128, 256, 512],
+                            n_input_channels=LEN_ABRV, num_classes=1)
+resnet18_full = nets.ResNet('basic', [2, 2, 2, 2], [64, 128, 256, 512],
+                            n_input_channels=LEN_FULL, num_classes=1)
+resnet50_full = nets.ResNet('bottleneck', [3, 4, 6, 3], [64, 128, 256, 512],
+                            n_input_channels=LEN_FULL, num_classes=1)
+densenet_abrv = nets.DenseNet(spatial_dims=3, in_channels=LEN_ABRV, out_channels=1,)
+
+model_names = {"resnet18_single": resnet18_single,
+               "resnet50_single": resnet50_single,
+               "resnet18_abrv": resnet18_abrv,
+               "resnet50_abrv": resnet50_abrv,
+               "resnet18_full": resnet18_full,
+               "resnet50_full": resnet50_full
+                }

auto_detect_breast_mri/models/unet_crop.py

@@ -0,0 +1,290 @@
+"""The pretrained UNet that crops air and thoracic cavity off a breast MRI."""
+
+import gc
+
+import matplotlib.pyplot as plt
+import monai.losses
+import numpy as np
+import torch
+from monai.networks import nets
+from torch import nn
+from torch.utils.data import DataLoader
+
+
+class UNet_Crop(object):
+    def __init__(self, path_to_unet='', saving_path='unet.pth', input_channel=3, running_device='cpu',
+                 show_validation=False, criterion='dice_ce'):
+        """
+        Crop an input image from top and bottom so that air and thoracic cavity are cut to save computational costs.
+        Use a trained UNet for this
+        :param path_to_unet: str
+            defines the path to the definition of the pretrained UNet to be used for cropping. Must be a .pth file.
+            if empty, the network will be trained and saved to unet.pth
+        :param criterion: str
+            defines the criterion to be used while training. Can be one of ['dice_ce', 'dice', 'ce']
+        """
+        # init
+        if input_channel > 0:
+            self.in_channel = input_channel
+        else:
+            self.in_channel = 3
+        if running_device.__eq__('cpu') or running_device.__eq__('cuda'):
+            self.device = running_device
+        else:
+            raise AttributeError(
+                'Device {} is unknown. Please use an existing device (cpu or cuda).'.format(running_device))
+        unet_channel = [self.in_channel, 64, 128, 256, 512]
+        unet_strides = [2, 2, 2, 2]
+        # output: background and breast mask
+        self.cropping_unet = nets.Unet(3, self.in_channel, 2, unet_channel, unet_strides,
+                                       act='ReLU', adn_ordering="NAD")
+        self.show_validation_images = show_validation
+        print('Init Cropping Unet: done.')
+
+        if path_to_unet is not None and path_to_unet.endswith('.pth'):
+            checkpoint = torch.load(path_to_unet, weights_only=True)
+            self.cropping_unet.load_state_dict(checkpoint)
+            self.cropping_unet.eval()
+        if saving_path is not None and saving_path.endswith('.pth'):
+            self.best_path = saving_path.split('.pth')[0] + '_best.pth'
+            self.saving_path = saving_path
+
+        self.criterion = monai.losses.DiceCELoss(softmax=True, include_background=True)
+        self.set_criterion(criterion)
+
+    def __call__(self, image: torch.Tensor):
+        """
+        Crop the given image so that air and thoracic cavity are reduced
+        :param image:
+            image to be cropped in shape (B, M, W, H, D) where
+            the first dimension (batch) is optional,
+            M refers to the MRI modes used (Appreviated protocol: M=2, Full protocol: M=6)
+            D defines depth, H height, and W width.
+        :return:
+            the cropped image with output shape (B, M, W, H/2, D); in case image was a 4D tensor, B will be 1 here.
+            tuple with (min, max) where min and max define the index range kept after cropping
+            the breast mask
+        """
+        while len(image.shape) < 5:
+            wrapper = torch.empty((1,) + image.shape)
+            wrapper[0] = image
+            image = wrapper
+        if len(image.shape) != 5:
+            raise ValueError("Image must be a 5D Tensor or Tensor, which can be wrapped into 5D. But was {}D.".format(
+                len(image.shape)))
+        print("Input image shape to UNet_Crop: " + str(image.shape))
+        output = self.cropping_unet(image)
+        print("Output shape cropped image: " + str(output.shape))
+        # TODO actually we should compute breast mask for each input (output here) and not for all at once
+        mask, (w_min, w_max) = self.extract_breast_mask(output, int(output.shape[-2] / 2))
+        #if torch.is_tensor(w_min):
+        #    w_min = int(w_min.item())
+        #if torch.is_tensor(w_max):
+        #    w_max = int(w_max.item())
+        breast_mask = output[:, 1] - output[:, 0]  # torch.ge(output[:, 1], output[:, 0])  # binary output
+        #breast_mask = breast_mask[0] if len(breast_mask.shape) == 5 else breast_mask 
+        print("Breast mask: " + str(breast_mask.shape))
+        cropped_images = []
+        for b in range(image.shape[0]):
+            w_min_b = int(w_min[b].item())
+            w_max_b = int(w_max[b].item())
+            cropped_images.append(image[b, :, :, w_min_b:w_max_b, :])
+
+        cropped_images = torch.stack(cropped_images, dim=0)
+        return cropped_images, (w_min, w_max), breast_mask
+
+    def get_parameters(self):
+        return self.cropping_unet.parameters()
+
+    def set_criterion(self, criterion):
+        if criterion is not None:
+            if criterion == 'ce':
+                self.criterion = nn.CrossEntropyLoss()
+            elif criterion == 'dice':
+                self.criterion = monai.losses.DiceLoss(softmax=True, include_background=True)
+            elif criterion == 'dice_ce':
+                self.criterion = monai.losses.DiceCELoss(softmax=True, include_background=True)
+            else:
+                print('Criterion is unknown: {}. Expected any of [\'dice_ce\', \'dice\', \'ce\']. '
+                      'Set to dice_ce as default.'.format(criterion))
+        else:
+            print('Criterion is None. Set to dice_ce as default.'.format(criterion))
+
+    def validate(self, n_epochs: int, validation_loader: DataLoader):
+        self.cropping_unet.eval()
+        depth = 15  # depth of MRI scan to visualize for validation
+        for epoch in range(n_epochs):
+            with torch.no_grad():
+                val_loss = 0
+                gc.collect()
+                # Calculate validation loss
+                for val_batch, item in enumerate(validation_loader):
+                    if item.__class__() == dict().__class__():
+                        # dataloader
+                        val_target = item['image']['data']
+                        val_mask = item['label']['data']
+                    else:
+                        # subjects loader
+                        val_target = item.image.data
+                        val_mask = item.label.data
+
+                    # send to correct device
+                    val_target.to(self.device)
+                    val_mask.to(self.device)
+
+                    # compute the mask
+                    val_probs = self.cropping_unet(val_target)
+
+                    if self.show_validation_images and val_batch % 10 == 0:
+                        cropped_mask, (c_min, c_max) = self.extract_breast_mask(val_probs, int(val_mask.shape[-1] / 2))
+
+                        cropped_image = val_target[0, 0, depth, int(c_min[0].item()):int(c_max[0].item()), :]
+                        plt.imshow(torch.cat([val_target[0][0][depth],
+                                              val_mask[0][0][depth]]), cmap='gray', interpolation='none')
+                        plt.show()
+                        plt.imshow(torch.cat([cropped_image, cropped_mask[0][depth]]), cmap='gray',
+                                   interpolation='none')
+                        plt.show()
+
+                    loss = self.criterion(val_probs[:, 1, :, :, :][:, None], val_mask)
+                    val_loss += loss.item()
+            print("Epoch [{}/{}] | UNet - Average Validation Loss: {:.5f}".format(epoch + 1, n_epochs,
+                                                                                  val_loss / len(validation_loader)))
+
+    def train_unet(self, n_epochs, train_loader, optimizer):
+        self.cropping_unet.train(True)
+        best_loss = torch.inf
+        for epoch in range(n_epochs):
+            print('EPOCH {}:'.format(epoch + 1))
+            train_loss = 0
+            for batch_id, item in enumerate(train_loader):
+                if item.__class__() == dict().__class__():
+                    # dataloader
+                    data = item['image']['data']
+                    mask = item['label']['data']
+                else:
+                    # subjects loader
+                    data = item.image.data
+                    mask = item.label.data
+
+                data.to(self.device)
+                mask.to(self.device)
+                optimizer.zero_grad()
+
+                #pred_probs = self.cropping_unet(data)
+                (cropped_image, cropping_bounds, breast_mask) = self.__call__(data)
+                # intermediate validation
+                if (batch_id % 10) == 0 or batch_id == 0:
+                    print(torch.cat([data[0][0][10], mask[0][0][10]], dim=-1).shape)
+                    # TODO fix dimension issue
+                    plt.imshow(torch.cat([data[0][0][10], mask[0][0][10]], dim=-1), cmap='gray', interpolation='none')
+                    plt.show()
+                    plt.imshow(breast_mask[0][10].detach().numpy(), cmap='gray', interpolation='none')
+                    plt.show()
+                
+                breast_mask = np.expand_dims(breast_mask.detach().numpy(), axis=1)
+                print("Subject data received by train loader:")
+                print(data.shape)
+                print("Subject label received by train loader:")
+                print(mask.shape)
+                print("UNet cropped data:")
+                print(breast_mask.shape)
+                
+                loss = self.criterion(breast_mask, mask)
+                loss.backward()
+
+                optimizer.step()
+
+                train_loss += loss.item()
+            if train_loss / len(train_loader) < best_loss:
+                self.save_unet(self.best_path)
+
+            if epoch % n_epochs / 5 == 0 or epoch == n_epochs - 1:
+                self.save_unet(self.saving_path)
+
+        self.cropping_unet.train(False)
+
+    def save_unet(self, path: str):
+        """
+        Save this unet to specified path. Otherwise, it is saved to default location unet.pth
+        :param path:
+        :return:
+        """
+        if path is not None and len(path) > 0:
+            torch.save(self.cropping_unet.state_dict(), path)
+        else:
+            torch.save(self.cropping_unet.state_dict(), 'unet.pth')
+
+    def extract_breast_mask(self, probability_output: torch.Tensor, mask_width: int):
+        """
+        Extract the breast mask from the given probability output tensor in which
+        the first dimension holds probabilities for each position to be background and
+        the second dimension holds those to be breast.
+        :param probability_output:
+            tensor of shape 2 x height x width x depth
+        :param mask_width:
+            output width for the breast mask. In case this defines a larger space than needed from the actual breast,
+            actual breast is found centered in the output mask within the according dimension.
+            If mask_width is smaller than the span of pixels defining the breast, the center of the breast is calculated
+            used to center the mask.
+            In other words, this parameter can be used to crop the mask to a certain width if it is a positive number.
+        :return:
+            3D tensor defining breast mask with according width if parameter is provided (depth x height/2 x mask_width)
+            otherwise output is of shape depth x height/2 x width
+        """
+        input_shape = probability_output.shape
+        if len(input_shape) == 4:
+            probability_output = probability_output.unsqueeze(0)
+            input_shape = probability_output.shape
+        if len(probability_output.shape) != 5:
+            raise ValueError(f"Invalid input shape. Expected 4D or 5D tensor, but got {probability_output.shape}")
+
+        breast_mask = torch.ge(probability_output[:, 1], probability_output[:, 0])
+        if 0 < mask_width < input_shape[-2]:
+            # get indices of values that are not 0
+            breast_indices = torch.tensor(
+                [[[[j for k in range(input_shape[-1])] for j in range(input_shape[-2])] for i in
+                  range(input_shape[-3])] for h in range(input_shape[0])])
+            breast_indices[breast_mask == 0] = 0
+
+            out_batch, out_width, out_height, out_depth = input_shape[0], input_shape[2], int(input_shape[3] / 2), \
+                input_shape[4]
+            batch_crop_range = torch.zeros(2, out_batch)
+            cropped_breast_mask = torch.zeros((out_batch, out_width, out_height, out_depth))
+            # compute center for each batch and each depth
+            for b in range(out_batch):
+                centers_per_depth = []
+                for d in range(out_depth):
+                    # crop width of the mask in this depth
+                    center_indices = []
+                    for col in range(out_width):
+                        indices_sum = sum(breast_indices[b, col, :, d])
+                        if indices_sum > 0:  # only count columns where a center could be calculated
+                            len_non_zero_indices = torch.sum(torch.where(breast_indices[b, col, :, d] > 0, 1.0, 0.0))
+                            center_indices.append(indices_sum / len_non_zero_indices)
+                    if len(center_indices) > 0:
+                        centers_per_depth.append(sum(center_indices) / len(center_indices))
+
+                # calculate center of this batch over all depth slices
+                hor_center = out_height / 2
+                if len(centers_per_depth) > 0:
+                    hor_center = (sum(centers_per_depth) / len(centers_per_depth)).item()
+                hor_min = round(hor_center - mask_width / 2)  # horizontal minimum
+                hor_max = round(hor_center + mask_width / 2)  # horizontal maximum
+
+                # ensure that cropping area is valid
+                if hor_min < 0:
+                    hor_min = 0
+                    hor_max = mask_width
+                elif hor_max > input_shape[-2]:
+                    hor_min = input_shape[-2] - mask_width
+                    hor_max = input_shape[-2]
+
+                batch_crop_range[0][b] = hor_min
+                batch_crop_range[1][b] = hor_max
+                cropped_breast_mask[b] = breast_mask[b, :, hor_min:hor_max, :]
+            return cropped_breast_mask, (batch_crop_range[0], batch_crop_range[1])
+        else:
+            return breast_mask.float(), (0, input_shape[-2])
+
+

auto_detect_breast_mri/preprocessing/__init__.py

@@ -0,0 +1 @@
+"""Turning raw DICOM/NIfTI archives into the layout the datasets expect."""

auto_detect_breast_mri/preprocessing/anonymize_dicom.py

@@ -0,0 +1,288 @@
+import os
+import shutil
+
+import pandas as pd
+import pydicom
+
+
+def anonymize(source_folder: str, destination_folder: str, mapping_file: str):
+    """
+    ====================
+    Anonymize DICOM data
+    ====================
+
+    This example is a starting point to anonymize DICOM data.
+
+    It shows how to read data and replace tags: person names, patient id,
+    optionally remove curves and private tags, and write the results in a new file.
+
+    """
+
+    # authors : Guillaume Lemaitre <g.lemaitre58@gmail.com>
+    # license : MIT
+    tags_to_remove = ["InstanceCreationDate", "InstanceCreationTime", "StudyDate", "SeriesDate",
+                      "AcquisitionDate", "ContentDate",
+                      "OverlayDate", "CurveDate", "PatientsWeight" "AcquisitionDatetime", "StudyTime", "SeriesTime",
+                      "AcquisitionTime", "ContentTime", "OverlayTime", "CurveTime",
+                      "InstitutionName", "InstitutionAddress", "ReferringPhysiciansName", "ReferringPhysician'sAddress",
+                      "ReferringPhysiciansTelephoneNumbers", "ReferringPhysicianIdentificationSequence",
+                      "CodeValue", "CodingSchemeDesignator", "CodeMeaning", "StationName", "StudyDescription",
+                      "ProcedureCodeSequence", "InstitutionalDepartmentName", "PhysiciansofRecord",
+                      "PhysiciansofRecordIdentificationSequence", "PerformingPhysiciansName",
+                      "PerformingPhysicianIdentificationSequence", "NameofPhysiciansReadingStudy",
+                      "PhysiciansReadingStudyIdentificationSequence",
+                      "OperatorsName", "AdmittingDiagnosesDescription", "ManufacturerModelName",
+                      "ReferencedPerformedProcedureStepSequence", "ReferencedImageSequence", "DerivationDescription",
+                      "IssuerofPatientID", "PatientBirthTime",
+                      "PatientBirthDateInAlternativeCalendar", "PatientsDeathDateInAlternativeCalendar",
+                      "PatientsAlternativeCalendar", "PatientSex",
+                      "OtherPatientIDs", "OtherPatientNames", "OtherPatientIDsSequence", "PatientBirthName",
+                      "PatientAge", "PatientSize", "PatientWeight", "PatientAddress", "PatientMothersBirthName",
+                      "MedicalRecordLocator", "MedicalAlerts", "ContrastAllergies", "CountryofResidence",
+                      "RegionofResidence", "PatientTelephoneNumbers", "EthnicGroup", "Occupation",
+                      "AdditionalPatientHistory",
+                      "PregnancyStatus", "ResponsibleOrganization", "PatientComments", "ClinicalTrialSponsorName",
+                      "ClinicalTrialProtocolID", "ClinicalTrialProtocolName", "ClinicalTrialSiteID",
+                      "ClinicalTrialSiteName",
+                      "ClinicalTrialSubjectID", "ClinicalTrialSubjectReadingID", "ClinicalTrialTimePointID",
+                      "ClinicalTrialTimePointDescription", "ClinicalTrialCoordinatingCenterName", "BodyPartExamined",
+                      "DeviceSerialNumber",
+                      "ImageComments", "RequestingPhysician", "RequestingService", "RequestedProcedureDescription",
+                      "RequestedContrastAgent", "StudyComments", "SpecialNeeds",
+                      "CurrentPatientLocation", "PatientsInstitutionResidence", "PatientState",
+                      "ScheduledPerformingPhysiciansName", "PerformedStationAETitle", "PerformedStationName",
+                      "PerformedLocation",
+                      "PerformedProcedureStepStartDate", "PerformedProcedureStepStartTime",
+                      "PerformedProcedureStepEndDate", "PerformedProcedureStepEndTime", "PerformedProcedureStepStatus",
+                      "PerformedProcedureStepID",
+                      "PerformedProcedureStepDescription", "PerformedProcedureTypeDescription",
+                      "PerformedProtocolCodeSequence", "RequestAttributesSequence",
+                      "CommentsonthePerformedProcedureStep", "RequestedProcedureID",
+                      "ReasonfortheRequestedProcedure", "RequestedProcedurePriority", "PatientTransportArrangements",
+                      "RequestedProcedureLocation", "RequestedProcedureComments", "ReasonfortheImagingServiceRequest",
+                      "IssueDateofImagingServiceRequest", "IssueTimeofImagingServiceRequest", "OrderEnterersLocation",
+                      "OrderCallbackPhoneNumber", "ImagingServiceRequestComments", "ConceptNameCodeSequence",
+                      "VerifyingObserverSequence",
+                      "DateTime", "Date", "Time", "PersonName", "InstanceCreatorUID", "UID", "ContentSequence", "SomeTag"]
+
+    ###############################################################################
+    # Anonymize a single file
+    ###############################################################################
+
+    mapping_df = pd.read_excel(mapping_file)
+
+    for source_id in mapping_df['UKA Study ID']:
+        # Create destination folder with according ID from mapping_df
+        dest_id = mapping_df[mapping_df['UKA Study ID'] == source_id]['ID'].iloc[0]
+        dest_pat_root = destination_folder + os.sep + dest_id
+        os.makedirs(dest_pat_root, exist_ok=True)
+
+        source_pat_root = source_folder + os.sep + str(source_id)
+        for root, dirs, files in os.walk(source_pat_root):
+            # folders with dyn: 'MR NCIdyn sense1.5', 'MR NCIdyn sense030413-2', 'MR sNCIdyn DEV', 'MR NCIdyn sense030413 SENSE', 'MR NCIdyn DEV', 'MR NCIdyn sense030413'
+            # folders with T2: 'MR NCIT2TSE HR ckbk', 'MR NCIT2TSE HR ckbk-2', 'MR T2 COR Silikon', 'MR T2 COR DEV', 'MR NCIT2TSE HR', 'MR NCIT2TSE HR ckbk-3', 'MR NCIT2TSE HR-2'
+            current_folder = root.split(os.sep)[-1]
+            if (('NCIdyn' in current_folder or 'MR Dyn' in current_folder or
+                'sNCIdyn' in current_folder or 'MR sDyn' in current_folder or
+                'MR MIP Orth' in current_folder or
+                'T2' in current_folder) and len(files) > 1):
+                stored_dirs = os.listdir(dest_pat_root)
+                if current_folder[-2] == '-':
+                    # there are several versions of this sequence. Take the latest one.
+                    prev_version_dir = None
+                    if current_folder[:-2] in stored_dirs:
+                        prev_version_dir = dest_pat_root + os.sep + current_folder[:-2]
+                    if current_folder[:-1] + str(int(current_folder[-1]) - 1) in stored_dirs:
+                        prev_version_dir = dest_pat_root + os.sep + current_folder[:-1] + str(
+                            int(current_folder[-1]) - 1)
+                    if prev_version_dir is not None:
+                        shutil.rmtree(prev_version_dir)
+                dest_root = dest_pat_root + os.sep + current_folder
+                os.makedirs(dest_root, exist_ok=True)
+
+                scanning_sequence = None
+                for file in files:
+                    dataset = pydicom.dcmread(root + os.sep + file)
+
+                    ###############################################################################
+                    # We can define a callback function to find all tags corresponding to a person
+                    # names inside the dataset. 
+
+                    def person_names_callback(dataset, data_element):
+                        if data_element.VR == "PN":
+                            data_element.value = "anonymous"
+
+                    ###############################################################################
+                    # We can use the callback function to iterate through the dataset but
+                    # also some other tags such that patient ID, etc.
+
+                    dataset.PatientID = dest_id
+                    if dest_id != dest_id or dataset.PatientID != dataset.PatientID or len(dest_id) < 1:
+                        print("PatientID was not set for " + root + os.sep + file)
+                    dataset.walk(person_names_callback)
+
+                    ###############################################################################
+                    # pydicom allows to remove private tags using ``remove_private_tags`` method
+
+                    dataset.remove_private_tags()
+
+                    ###############################################################################
+                    # Anonymize some specific parameter
+
+                    tag = 'PatientBirthDate'
+                    if tag in dataset:
+                        dataset.data_element(tag).value = '19000101'  # allow-private-path: DICOM date placeholder
+
+                    tag = 'AccessionNumber'
+                    if tag in dataset:
+                        dataset.data_element(tag).value = dest_id
+
+                    tag = 'StudyID'
+                    if tag in dataset:
+                        dataset.data_element(tag).value = ''
+
+                    if 'ScanningSequence' in dataset:
+                        scanning_sequence = dataset.data_element("ScanningSequence")
+                    elif scanning_sequence is not None:
+                        # try to set Scanning sequence according to previous slices
+                        dataset.add_new(scanning_sequence.tag, scanning_sequence.VR, scanning_sequence.value)
+
+                    ###############################################################################
+                    # Delete all tags that are not required
+                    deleted_tags = 0
+                    for tag in tags_to_remove:
+                        if tag in dataset:
+                            deleted_tags = deleted_tags + 1
+                            delattr(dataset, tag)
+                    #print(f"Deleted {deleted_tags} out of {len(tags_to_remove)} tags.")
+                    ##############################################################################
+                    # Finally, store the image
+                    dataset.save_as(dest_root + os.sep + file)
+
+
+def keep_four_folders(destination_folder):
+    # check that there are only four folders for each UK_n: 1: dyn folder with most files, 2: T2 folder with most files
+    for root, dirs, files in os.walk(destination_folder):
+        if root.split(os.sep)[-1].startswith('UKA_'):
+            if len(dirs) > 2:
+                # check if there is too many folders for dynamic or t2 weighted sequence
+                dyn_folder = []
+                sub_folder = []
+                t2_folder = []
+                for dir in dirs:
+                    if 'SDYN' in dir.upper() or 'SNCIDYN' in dir.upper():
+                        sub_folder.append(dir)
+                    elif 'DYN' in dir.upper():
+                        dyn_folder.append(dir)
+                    elif 'T2' in dir.upper():
+                        t2_folder.append(dir)
+                    else:
+                        print('Folder {} neither recognized as dynamic nor as T2 sequence folder.'.format(dir))
+                # remove the folder with less files
+                if len(dyn_folder) > 1:
+                    prev_dyn = dyn_folder[0]
+                    for dyn in dyn_folder:
+                        if len(os.listdir(root + os.sep + dyn)) < len(os.listdir(root + os.sep + prev_dyn)):
+                            shutil.rmtree(root + os.sep + dyn)
+                        else:
+                            if dyn is not prev_dyn:
+                                shutil.rmtree(root + os.sep + prev_dyn)
+                                prev_dyn = dyn
+                if len(sub_folder) > 1:
+                    pre_sub = sub_folder[0]
+                    for sub in sub_folder:
+                        if len(os.listdir(root + os.sep + sub)) < len(os.listdir(root + os.sep + pre_sub)):
+                            shutil.rmtree(root + os.sep + sub)
+                        else:
+                            if sub is not pre_sub:
+                                shutil.rmtree(root + os.sep + pre_sub)
+                                pre_sub = sub
+                if len(t2_folder) > 1:
+                    prev_t2 = t2_folder[0]
+                    for t2 in t2_folder:
+                        if len(os.listdir(root + os.sep + t2)) < len(os.listdir(root + os.sep + prev_t2)) or ("COR" in t2 and t2 is not prev_t2):
+                            # remove the folder with less files (do not use folders with COR)
+                            shutil.rmtree(root + os.sep + t2)
+                        else:
+                            if t2 is not prev_t2:
+                                shutil.rmtree(root + os.sep + prev_t2)
+                                prev_t2 = t2
+
+            elif len(dirs) < 2:
+                print('Not enough folder for patient {}'.format(root))
+
+
+if __name__ == "__main__":
+    """
+    Anonymize a DICOM tree and normalise its sequence folder names.
+
+        python -m auto_detect_breast_mri.preprocessing.anonymize_dicom <source_folder> <destination_folder> <mapping_file.xlsx> \
+            --output-folder <folder for the renamed tree>
+
+    All paths must be given explicitly: this script touches patient data, so it must never carry a
+    built in default.
+    """
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Anonymize a DICOM tree and rename its sequences.")
+    parser.add_argument("source_folder", help="DICOM tree to read")
+    parser.add_argument("destination_folder", help="where the anonymized tree is written")
+    parser.add_argument("mapping_file", help="xlsx mapping original IDs to pseudonyms")
+    parser.add_argument("--output-folder", default=None,
+                        help="where the sequence-renamed tree is written. "
+                             "Default: <destination_folder>_renamed")
+    args = parser.parse_args()
+
+    source_folder = args.source_folder
+    destination_folder = args.destination_folder
+    mapping_file = args.mapping_file
+    anonymize(source_folder, destination_folder, mapping_file)
+    keep_four_folders(destination_folder)
+
+    output_folder = args.output_folder or (destination_folder.rstrip(os.sep) + "_renamed")
+    sequence_mapping = {'MR sNCIdyn DEV': 'Sub_T1',
+                        'MR NCIdyn sense1.5': 'Dynamic_T1',
+                        'MR NCIT2TSE HR ckbk': "T2"}
+
+    # sanitize remaining files and rename folders for consistency:
+    for root, dirs, files in os.walk(destination_folder):
+        uid_folder = root.replace(destination_folder, output_folder)
+        sequence = uid_folder.split(os.sep)[-1]
+        if sequence in sequence_mapping.keys():
+            sub_folder = uid_folder.replace(sequence, sequence_mapping.get(sequence))
+        elif sequence.startswith('MR NCIdyn') or sequence.startswith('MR Dyn'):
+            sub_folder = uid_folder.replace(sequence, 'Dynamic_T1')
+            print("Mapped sequence {sequence} to Dynamic_T1 for {root}".format(sequence=sequence, root=root))
+        elif sequence.startswith('MR sDyn'):
+            sub_folder = uid_folder.replace(sequence, 'Sub_T1')
+            print("Mapped sequence {sequence} to Sub_T1 for {root}".format(sequence=sequence, root=root))
+        elif sequence.startswith('MR NCIT2') or sequence.startswith("MR T2"):
+            sub_folder = uid_folder.replace(sequence, 'T2')
+            print("Mapped sequence {sequence} to T2 for {root}".format(sequence=sequence, root=root))
+        else:
+            sub_folder = uid_folder
+        os.makedirs(sub_folder, exist_ok=True)
+        for file in files:
+            dataset = pydicom.dcmread(root + os.sep + file)
+            if len(dataset.PatientID) < 1:
+                print("Patient ID was not set for " + root + os.sep + file)
+            shutil.copy(root + os.sep + file, sub_folder)
+
+    '''for root, dirs, files in os.walk(output_folder):
+        sequence_name = root.split(os.sep)[-1]
+        if 'SNCIDYN' in sequence_name.upper():
+            renamed_folder = root.replace(sequence_name, 'Sub_T1')
+            shutil.move(root, renamed_folder)
+            shutil.rmtree(root)
+            print(f"Renamed {sequence_name} to Sub_T1 in {root}")
+        elif 'DYN' in sequence_name.upper():
+            renamed_folder = root.replace(sequence_name, 'Dynamic_T1')
+            shutil.move(root, renamed_folder)
+            shutil.rmtree(root)
+            print(f"Renamed {sequence_name} to Dynamic_T1 in {root}")
+        elif 'T2' in sequence_name.upper():
+            renamed_folder = root.replace(sequence_name, 'T2')
+            shutil.move(root, renamed_folder)
+            shutil.rmtree(root)
+            print(f"Renamed {sequence_name} to T2 in {root}")'''
+

auto_detect_breast_mri/preprocessing/split_breasts.py

@@ -0,0 +1,337 @@
+import os
+from pathlib import Path
+
+import pandas
+import torchio as tio
+
+from auto_detect_breast_mri.data.transforms import *
+from auto_detect_breast_mri.models.unet_crop import UNet_Crop
+import nibabel as nib
+
+class Preprocessor:
+    """
+    Preprocessing of Breast MRI data to be expected in a hierarchical folder structure within path_root.
+    All images_full_data are cropped according to the given pretrained UNet, normalized, split into right and left half and saved.
+    :param path_root: str
+        defines the path to the root folder in which the MRI images_full_data to be loaded into this dataset are stored.
+        path_root can either be
+        a single folder [Duke-Breast-Cancer-MRI] with folders [001, 002, ..., 00n]
+        or a pkl-file from which pickled dataset data is loaded. (in this case, path_features must also be a pkl file.)
+    :param shape:
+        5D tensor specifying (number of sample loaded * 2, channel, depth, height, width).
+        Number of samples is the number of folders named Breast_MRI_xxn. It is multiplied by 2 in shape because images_full_data
+        are split into left and right half.
+    :param cropping_net
+        defines the pretrained model used for cropping if cropping shall be performed. Must be a UNet_Crop
+    :param protocol: list of strings
+        all protocol files for which preprocessing shall be performed 'pre'-images_full_data are always processed.
+        must be one or several of
+        'post_1', 'post_2', 'post_3', 'post_4', or 'T1'
+        default is all of the above.
+    """
+
+    def __init__(self, path_root: str,
+                 path_feature: str,
+                 cropping_net,
+                 shape=(32, 512, 512),
+                 target_spacing=(0.7, 0.7, 3),
+                 protocol=['pre', 'post_1', 'post_2', 'post_3', 'post_4', 'T2'],
+                 post_prefix="post_"):
+        self.available_protocols = ['pre', 'post_1', 'post_2', 'post_3', 'post_4', 'T2_resampled', 'sub']
+        self.prefix = 'post_'
+        if not self.prefix.__eq__(post_prefix):
+            self.available_protocols[0] = post_prefix + '0'
+            self.available_protocols[-1] = 'Sub'
+            for i, prot in enumerate(self.available_protocols):
+                if prot.__contains__(self.prefix):
+                    self.available_protocols[i] = self.available_protocols[i].replace(self.prefix, post_prefix)
+            self.prefix = post_prefix
+        # init parameters
+        if path_root:
+            self.path_root = Path(path_root)
+        else:
+            raise AttributeError('A None root path can not instance a dataset object. '
+                                 'Please specify the path where to find Breast Cancer MRIs from the Duke Dataset.')
+        if path_feature and len(path_feature):
+            self.path_feature = Path(path_feature)
+        else:
+            raise AttributeError('A None feature path can not instance a dataset object. '
+                                 'Please specify the path where to find labels for this dataset.')
+        if shape and all(dim > 0 for dim in shape):
+            (self.depth, self.height, self.width) = shape
+        else:
+            (self.depth, self.height, self.width) = (32, 448, 448)
+        if target_spacing and all(dim > 0 for dim in target_spacing):
+            self.target_spacing = target_spacing
+        else:
+            self.target_spacing = (0.7, 0.7, 3)
+        self.right_split = tio.Crop((int(self.width / 2), 0, 0, 0, 0, 0))
+        self.left_split = tio.Crop((0, int(self.width / 2), 0, 0, 0, 0))
+
+        if protocol:
+            valid_protocol = [True if val in self.available_protocols else False for val in protocol]
+            if all(valid_protocol):
+                if not protocol.__contains__(self.available_protocols[0]):
+                    self.mri_protocol = [self.available_protocols[0]]
+                    for prot in protocol:
+                        self.mri_protocol.append(prot)
+                else:
+                    self.mri_protocol = protocol
+            else:
+                print(
+                    '{} is not a protocol supported. Please check. For now the following sequences are considered: {}'.format(
+                        protocol[valid_protocol.index(False)], self.available_protocols))
+                self.mri_protocol = self.available_protocols
+
+        self.in_channel = 1 if self.mri_protocol.__contains__('Sub') else len(self.mri_protocol)
+
+        # shape of a 3D half (right or left) of a breast MRI
+        self.single_half_shape = (self.in_channel, self.depth, int(self.height / 2), self.width)
+
+        if cropping_net is not None:
+            self.cropping_unet = cropping_net
+        else:
+            raise ValueError('Cropping UNet can not be None.')
+
+        if os.path.exists(self.path_root):
+            #malignity = pandas.read_excel(path_feature)
+            self.file_label_mapping = {'Path': [], 'Label': []}
+            if path_feature.endswith('.xlsx') or path_feature.endswith('.xls'):
+                self.label_container = pandas.read_excel(path_feature)
+            elif path_feature.endswith('.csv'):
+                self.label_container = pandas.read_csv(path_feature)
+            else:
+                raise ValueError('File type not supported.')
+
+            for c_channel, (root, _, files) in enumerate(os.walk(self.path_root)):
+                self.run_preprocessing_pipeline(root, files, self.label_container)
+                print(root + ' is done.')
+
+            # save filepath -  label mapping as pickled file
+            io_saver_loader.Pickle_Loader.store_pickle(self.file_label_mapping,
+                                                       feature_path + 'filepath_label_mapping.pkl')
+
+        else:
+            raise FileNotFoundError('Path root does not exist: ' + str(self.path_root)
+                                    + '. Dataset could not be initialized.')
+
+    def run_preprocessing_pipeline(self, pat_folder, nifti_files, malignity):
+        """
+        Run preprocessing pipeline for all nifti files within pat_folder.
+        The preprocessing pipeline includes:
+        reading nifti files, crop, normalize, split into right and left half, save images_full_data
+        According mapping between file paths and malignity labels is appended to this class dictionary
+        :param pat_folder: str
+            defines the path to the current root folder in which patient files might be found
+            this name is equal to the patient ID
+        :param nifti_files: list
+            all nii.gz-file names within root, might be empty if there are no such files
+        :param malignity: numpy.DataFrame
+            contains 2 columns:
+            First contains patientID concatenated with _ and one out of [left, right], e.g. 001_left
+            Second contains the actual label for this breast [True, False]
+            If there are no labels for this patient ID as in pat_folder, those nifti_files are skipped.
+        :return:
+        """
+        pat_ID = pat_folder.split(os.sep)[-1]
+        crop_range = int(self.width * 0.25), int(self.width * 0.75), int(self.width * 0.25), int(self.width * 0.75)
+        filenames = []
+        left_halves = []
+        right_halves = []
+        path_sceleton = pat_ID + '/{}'
+        out_pat_folder = pat_folder.replace(replacing_folder, 'UKA_cropped_sanitized')
+        out_root = out_pat_folder.replace(pat_ID, '')
+        out_folder_exists = Path(out_pat_folder).exists()
+
+        pre_filename = 'Dyn_0.nii.gz'
+        if not out_folder_exists or (out_folder_exists and len(os.listdir(out_pat_folder)) < len(nifti_files)):
+            reference_img = tio.Resample(self.target_spacing)(tio.ScalarImage(pat_folder + os.sep + pre_filename))
+            transform = tio.Compose([
+                tio.Resample(reference_img),
+                tio.CropOrPad((self.height, self.width, self.depth), padding_mode=0),
+                tio.ToCanonical()
+            ])
+
+            # process pre file first
+            if pre_filename in nifti_files:
+                filepath = os.path.join(pat_folder, pre_filename)
+                (pre_left, pre_right), crop_range = self.perform_preprocessing(filepath, transform, None, reference_img)
+                assert len(pre_left.data.unique()) > 1  # ensure that there is no image in one color
+                assert len(pre_right.data.unique()) > 1
+
+                filenames.append(pre_filename)
+                left_halves.append(pre_left)
+                right_halves.append(pre_right)
+
+                calc_sub_image = True
+            else:
+                calc_sub_image = False
+
+            # duplication is needed to ensure that pre.nii.gz is found before
+            for c_depth, filename in enumerate(nifti_files):
+                if str(filename).endswith('.nii.gz') and filename != pre_filename:
+                    mode = filename.split('.nii.gz')[0]
+                    filepath = os.path.join(pat_folder, filename)
+                    # crop all other files accordingly
+                    if mode.startswith('mask_breast') or mode in self.mri_protocol:
+                        (left_half, right_half), _ = self.perform_preprocessing(filepath, transform, crop_range,
+                                                                                reference_img)
+                        filenames.append(filename)
+
+                        if len(left_half.data.unique()) > 1 and len(right_half.data.unique()) > 1:
+                            left_halves.append(left_half)
+                            right_halves.append(right_half)
+
+                            # create subtraction image
+                            if calc_sub_image and mode.startswith(self.prefix):
+                                sub_reference_img = tio.Resample(self.target_spacing)(pre_left)
+                                sub_transform = tio.Compose([
+                                    tio.Resample(sub_reference_img),
+                                    tio.CropOrPad((int(self.height / 2), int(self.width / 2), self.depth),
+                                                  padding_mode=0),
+                                    tio.ToCanonical()
+                                ])
+                                # calc subtraction image
+                                sub_left = left_half.data - pre_left.data
+                                sub_right = right_half.data - pre_right.data
+                                sub_filename = path_sceleton + mode.replace(self.prefix, 'Sub') + '.nii.gz'
+                                self.save_as_ScalarImage(sub_left, sub_right, out_root, sub_filename, sub_transform,
+                                                         sub_reference_img.affine)
+                        else:
+                            tmp = "Right" if len(left_half.data.unique()) > 1 else "Left"
+                            print(tmp + " half of " + str(filepath) + " is empty.")
+
+            if len(left_halves) > 0 and len(right_halves) > 0:
+                self.save_scalar_images_and_mapping(left_halves, right_halves,
+                                                    out_root, path_sceleton, filenames)
+
+    def perform_preprocessing(self, path: str, transform, crop_range: tuple | None, reference_img: tio.ScalarImage):
+        """
+        Read the nifti file at path, transform it according to given transform, split into left and right halves
+        :param path: str
+            File path to the nifti file to be cropped
+        :param crop_range: tuple
+            range to which width is cropped. if it is None, pretrained Unet of this class is used for cropping
+            otherwise crop_range is supposed to be tuple of length 4 -> (min_left, max_left, min_right, max_right)
+        :return:
+            two tuple of length two each:
+            first containing left and right preprocessed images_full_data as torchio.ScalarImage
+            second containing cropping range (this only differs from crop_range if it has been None.)
+            If path is not a valid file, this method returns None instead of the tuples
+        """
+        if Path(path).is_file():
+            # read nifti
+            nifti = tio.ScalarImage(path)
+            nifti = transform(nifti)
+            # split left, right
+            nifti_left = self.left_split(nifti)
+            nifti_right = self.right_split(nifti)
+
+            # crop
+            if crop_range is None:
+                # TODO change to use CropOrPad from torchio with segmentation mask from UNet
+                ''' 
+                left_mask = self.cropping_unet.__call__(nifti_left.data)
+                left = tio.Subject(
+                    image = nifti_left,
+                    cropping_mask = tio.ToCanonical()(tio.LabelMap(tensor=left_mask.to(torch.int64), affine=reference_img.affine))
+                )
+                crop_left = tio.CropOrPad((nifti_left.shape[0], int(nifti_left.shape[1]/2), nifti_left.shape[2]), mask_name='cropping_mask')
+                cropped_left = crop_left(left)
+
+                right_mask = self.cropping_unet.__call__(nifti_right.data)
+                right = tio.Subject(
+                    image = nifti_right,
+                    cropping_mask = tio.LabelMap(tensor=right_mask, affine=reference_img.affine)
+                )
+                crop_right = tio.CropOrPad((nifti_left.shape[0], int(nifti_left.shape[1]/2), nifti_left.shape[2]), mask_name='cropping_mask')
+                cropped_right = crop_right(right)
+                '''
+                _, (left_min, left_max) = self.cropping_unet.__call__(nifti_left.data)
+                _, (right_min, right_max) = self.cropping_unet.__call__(nifti_right.data)
+            else:
+                (left_min, left_max, right_min, right_max) = crop_range
+            cropped_left = tio.Crop((0, 0, left_min, nifti_left.data.shape[2] - left_max, 0, 0))(nifti_left)
+            cropped_right = tio.Crop((0, 0, right_min, nifti_right.data.shape[2] - right_max, 0, 0))(nifti_right)
+
+            return (cropped_left, cropped_right), (left_min, left_max, right_min, right_max)
+        return None, None
+
+    def save_scalar_images_and_mapping(self, left: torch.Tensor, right: torch.Tensor, path_root: str,
+                                       path_sceleton: str, filenames=[]):
+        n_images = len(left)
+        # left and right are tensors of 3D image tensors -> filenames holds one filename for each image.
+        for pos in range(n_images):
+            path = path_sceleton + filenames[pos]
+            # create folder if not existing
+            folder = path_sceleton.replace(path_sceleton.split(os.sep)[-1], '')
+            if not os.path.exists(path_root + folder):
+                os.makedirs(path_root + folder)
+
+            left[pos].save(path_root + path.format('L-'))
+            right[pos].save(path_root + path.format('R-'))
+
+    def save_as_ScalarImage(self, left: torch.Tensor, right: torch.Tensor,
+                            path_root: str, path_sceleton: str, transform, affine):
+        left_image = tio.ScalarImage(tensor=left, affine=affine)
+        left_image = transform(left_image)
+        right_image = tio.ScalarImage(tensor=right, affine=affine)
+        right_image = transform(right_image)
+
+        left_path = path_sceleton.format('L-')
+        right_path = path_sceleton.format('R-')
+        l_folder = path_root + left_path.replace(left_path.split(os.sep)[-1], '')
+        r_folder = path_root + right_path.replace(right_path.split(os.sep)[-1], '')
+
+        # create folder if not existing
+        if not os.path.exists(l_folder):
+            os.makedirs(l_folder)
+        if not os.path.exists(r_folder):
+            os.makedirs(r_folder)
+        # save images_full_data
+        left_image.save(path_root + left_path)
+        right_image.save(path_root + right_path)
+
+
+if __name__ == '__main__':
+    """
+    Crop every NIfTI under the image root with the trained cropping UNet and split it into a left
+    and a right breast volume.
+
+        python -m auto_detect_breast_mri.preprocessing.split_breasts --unet-path <unet_best.pth>
+
+    Paths not given on the command line come from the site config (see config.example.yaml).
+    """
+    import argparse
+
+    from auto_detect_breast_mri.config import resolve_path
+
+    parser = argparse.ArgumentParser(description="Crop and split breast MRIs into left/right halves.")
+    parser.add_argument("--unet-path", required=True, help="pth file of the trained cropping UNet")
+    parser.add_argument("--data-path", default=None,
+                        help="image root to preprocess in place. Config key: data_root")
+    parser.add_argument("--shape", type=int, nargs=3, default=(512, 512, 32),
+                        metavar=("HEIGHT", "WIDTH", "DEPTH"),
+                        help="shape every volume is cropped/padded to. Default: 512 512 32")
+    parser.add_argument("--save_cropped_image", action="store_true", default=False, help="If --save_cropped_image is specified, a debug image is printed for investigation of the split.")
+    args = parser.parse_args()
+
+    path_root = resolve_path(args.data_path, "data_root", "image root to preprocess")
+    CropNet = UNet_Crop(path_to_unet=args.unet_path, input_channel=1)
+    crop = tio.CropOrPad(tuple(args.shape), padding_mode=0) # crop or pad to same size because UNet_Crop crops by half size
+    canonic = tio.ToCanonical()
+    printed_debug = False
+    for c_channel, (root, _, files) in enumerate(os.walk(path_root)):
+        for file in files:
+            image_data = tio.ScalarImage(root + os.sep + file)
+            image_data = crop(image_data)
+            image_data = canonic(image_data)
+            cropped_image = CropNet.__call__(image_data.data)
+            if not printed_debug and args.save_cropped_image:
+                output_root = root.replace('nifti', 'outputs') 
+                os.makedirs(output_root, exist_ok=True)
+                output_path = f"{output_root + os.sep + file}"
+                print(cropped_image)
+                nib.save(nib.Nifti1Image(cropped_image[0].numpy().astype(np.float32), affine=np.eye(4)), output_path)
+                printed_debug = True
+                print(f"saved cropped image to: {output_path}")

auto_detect_breast_mri/training/__init__.py

@@ -0,0 +1 @@
+"""Training and evaluation loops, LR schedules and the shared CLI."""

auto_detect_breast_mri/training/cli.py

@@ -0,0 +1,65 @@
+"""The command line interface shared by the training and comparison scripts."""
+
+import argparse
+
+
+def generate_parser():
+    '''
+    Generate the parser for this project, which includes parameters for
+    batch size, number of epochs, patience, fraction of training data, learning rate, fold, augmentation,
+    model path for pretrained networks, and flags 'train' and 'mixed precision'
+    :return: the ArgumentParser.
+    '''
+    train_fraction = [0.05, 0.1, 0.25, 0.5, 1.0]  # use all fractions if none is specified
+    parser = argparse.ArgumentParser(
+        prog="aiMRI",
+        description="Train a certain ResNet to classify malignity of Breast MRIs."
+    )
+    # required arguments
+    parser.add_argument("model_name",
+                        help="Enter the models name to be used.", type=str) # e.g. "resnet50_full", "resnet18_abrv", "resnet18" (for comparison)
+    # The three paths are optional here and fall back to the site config (config.example.yaml);
+    # giving them on the command line works exactly as before and always wins.
+    parser.add_argument("data_path", nargs='?', default=None,
+                        help="Path to the root folder of the (to the size 32x256x256) cropped MRIs. "
+                             "Config key: data_root.")
+    parser.add_argument("feature_path", nargs='?', default=None,
+                        help="Path to .xlsx/.csv file that contains label information for the desired criteria and for "
+                             "each data in data_path. Config key: metadata_file.")
+    parser.add_argument("split_files_folder", nargs='?', default=None,
+                        help="Path to folder containing the per-fold split .csv files with examination ID, label "
+                             "and side (see README, 'Data contract'). Config key: split_root.")
+    # optional arguments
+    parser.add_argument("-a", "--augmentation", type=str, choices=['basic', 'intensity'])
+    parser.add_argument("-b", "--batch_size", type=int, help="Specifies batch size to be user. Default is 4.")
+    parser.add_argument("-c", "--test_fold", type=int)  # c for cross validation (t already used)
+    parser.add_argument("-d", "--validation_fold", type=int)  # inner cross validation
+    parser.add_argument("--dataloader", type=str,
+                        help="Type of dataloader. Only 'subjects' is supported.")
+    parser.add_argument("-e", "--evaluation", action="store_true",
+                        help="If tag is added, evaluation will be performed."
+                             "Otherwise, only test (or train and test) will be performed.")
+    parser.add_argument("-f", "--fraction_of_data", type=float, choices=train_fraction,
+                        help="Fraction of data to be used for training. Default is to run once for all possible fractions.")
+
+    parser.add_argument("-l", "--learning_rate", type=float,
+                        help="Learning rate to be used as initial value. Default is 0.0001.")
+    parser.add_argument("-m", "--mixed_precision", action="store_true")
+    parser.add_argument("-n", "--number_of_epochs", type=int,
+                        help="Number of epochs to used for training. Default is 80.")
+    parser.add_argument("--output_path", type=str, default=None,
+                        help="Folder that checkpoints, ROC plots and result files are written to. "
+                             "Config key: output_root.")
+    parser.add_argument("-o", "--model_path", type=str, help="Path to pth-file of the pretrained model. "
+                                                             "Architecture must match the one defined by model_name.")
+    parser.add_argument("-p", "--patience", type=int,
+                        help="Patience for early stopping. Must be smaller than number_of_epochs; Default is 0.4 * "
+                             "number_of_epochs.")
+    parser.add_argument("-s", "--scheduler", type=str,
+                        help="Specify here which Learning rate scheduler should be used: Linear, ReduceLROnPlateau, Step, Multistep, CoscineAnnealing")
+    parser.add_argument("-t", "--train", action="store_true",
+                        help="If tag is added, training will be performed. "
+                             "Otherwise, parameter model_path must be specified or testing will evaluate untrained model.")
+    return parser
+
+

auto_detect_breast_mri/training/loops.py

@@ -0,0 +1,300 @@
+"""
+One epoch of training and of evaluation over a torchio SubjectsLoader, plus the wandb/metric
+bookkeeping around them. Batches are Subject dicts: batch['image']['data'], batch['label'],
+batch['path'].
+"""
+
+import gc
+import linecache
+import time
+import tracemalloc
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pylab
+import torch
+import wandb
+from torch import nn
+from torchmetrics import AUROC, Accuracy, Precision, Specificity, ROC
+
+CPU = "cpu"
+GPU = "cuda"
+
+
+def display_top(snapshot, key_type='lineno', limit=10, title=""):
+    """
+    Helper function to see top processes consumin most of the RAM for the current timepoint.
+    Limit can be used to define the number of entries to see. Default is 10, so top 10 entries are displayed.
+    """
+    snapshot = snapshot.filter_traces((
+        tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
+        tracemalloc.Filter(False, "<unknown>"),
+    ))
+    top_stats = snapshot.statistics(key_type)
+
+    print(f"Top {limit} lines {title}")
+    for index, stat in enumerate(top_stats[:limit], 1):
+        frame = stat.traceback[0]
+        print("#%s: %s:%s: %.1f KiB"
+              % (index, frame.filename, frame.lineno, stat.size / 1024))
+        line = linecache.getline(frame.filename, frame.lineno).strip()
+        if line:
+            print('    %s' % line)
+
+    other = top_stats[limit:]
+    if other:
+        size = sum(stat.size for stat in other)
+        print("%s other: %.1f KiB" % (len(other), size / 1024))
+    total = sum(stat.size for stat in top_stats)
+    print("Total allocated size: %.1f KiB" % (total / 1024))
+
+
+def plot_auc(roc_dict, figure_path, suffix=None):
+    wandb_suffix = suffix[:-1] if suffix.endswith('_') else suffix
+    fpr, tpr, thresholds, auc = roc_dict["fpr"], roc_dict["tpr"], roc_dict["thresholds"], roc_dict["auc"]
+
+    # Plot tpr vs 1-fpr
+    fig, ax = pylab.subplots()
+    pylab.plot(fpr, tpr, 'b', label="AUC = %0.3f" % auc)
+    pylab.legend(loc='lower right')
+    pylab.plot()
+    pylab.plot([0, 1], [0, 1], 'r--')
+    pylab.xlim([0, 1])
+    pylab.ylim([0, 1])
+    pylab.ylabel('True Positive Rate')
+    pylab.xlabel('False Positive Rate')
+    pylab.title(f'Receiver Operating Characteristic - {wandb_suffix} ')
+    pylab.show()
+    best_index = np.argmax(tpr - fpr)
+    if wandb_suffix is not None:
+        wandb.log({
+            f"ROC/{wandb_suffix}": fig,
+            f"Specificity (Youden)/{wandb_suffix}": 1 - fpr[best_index],
+            f"Sensitivity (Youden)/{wandb_suffix}": tpr[best_index]
+        })
+    roc_dict['Spec [i]'] = fpr[best_index]
+    roc_dict['Sens [i]'] = tpr[best_index]
+    roc_dict['i'] = best_index
+    pylab.savefig(figure_path)
+    pylab.close(fig)
+    return roc_dict
+
+
+def train_epoch_subjects(model, optimizer, criterion, data_loader, device, values, suffix="train_", show_figs=False,
+                         scaler=None, title_suffix="_"):
+    """
+    Training loop for one epoch when using subjects loader. 
+    Similar to train_epoch but handles subjects that contain data itself, label and path.
+    :param show_figs
+        Set to true if you want to see figures of the transformed data received by this function.
+        This can be useful for debugging purposes.
+    """
+    start = time.time()
+    model.train()
+    torch.set_grad_enabled(True)
+    epoch_loss = 0
+    counter = -1
+    wandb_suffix = suffix[:-1] if suffix.endswith('_') else suffix
+
+    for batch in data_loader:
+        data = batch['image']['data']
+        ground_truth = batch['label']
+        path = batch['path']
+        counter += 1
+        gc.collect()
+        if counter % 10 == 0:
+            if show_figs:
+                snapshot = tracemalloc.take_snapshot()
+                display_top(snapshot=snapshot, title=f"bath {counter}")
+
+                # Create a figure and axis objects for depth dimension (images of size 256x256)
+                fig, axs = plt.subplots(4, 8, figsize=(12, 6))
+
+                # Plot each slice of the tensor
+                for i in range(4):
+                    for j in range(8):
+                        index = i * 8 + j
+                        axs[i, j].imshow(torch.clip(data[0, 0, index], 0, 100), cmap='gray', interpolation=None)
+                        axs[i, j].axis('off')
+                        title = path[0] if i == 0 and j == 0 else f"Slice {index}"
+                        axs[i, j].set_title(title)
+
+                # Adjust layout and display the plot
+                plt.tight_layout()
+                plt.show()
+                plt.close()
+
+        data = data.to(device)
+        if not torch.is_tensor(ground_truth):
+            ground_truth = torch.tensor(ground_truth)
+        ground_truth = ground_truth.to(device)
+        optimizer.zero_grad()
+
+        if scaler is not None:
+            with torch.autocast(device_type=device, dtype=torch.float16):
+                pred_probs = model(data.float())
+                loss = criterion(pred_probs[:, 0].float(), ground_truth.float())
+
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            pred_probs = model(data.float())
+            loss = criterion(pred_probs[:, 0].float(), ground_truth.float())
+            loss.backward()
+            optimizer.step()
+
+        if values is not None:
+            update_evaluation_metrics(suffix, pred_probs, ground_truth, values)
+
+        wandb.log({f"loss/{wandb_suffix}": loss.item()}, commit=False)
+        epoch_loss += loss.item()
+        print('Train step loss: {}'.format(loss.item()))
+
+    avg_epoch_loss = epoch_loss / len(data_loader)
+    end = time.time()
+    print('Train {}: Duration = {} sec\n\tAverage Loss: {:.6f}'.format(
+        title_suffix,
+        end - start,
+        avg_epoch_loss))
+
+    if values is not None:
+        log_evaluation(suffix, values, False)
+    return avg_epoch_loss
+
+
+def update_evaluation_metrics(suffix: str, pred_probs, ground_truth, values):
+    auc_roc, roc, acc, sens, spec = values
+    for metric in [auc_roc, roc, acc, sens, spec]:
+        metric[suffix].update(pred_probs[:, 0], ground_truth)
+
+
+def log_evaluation(suffix: str, values, plot_tpr_fpr_path=None):
+    '''
+    Log evaluation metrices with weights and biases. 
+    @param suffix: 
+        suffix is the key for all dictionaries of the metrices in values. Also it will be used as suffix for the wandb logging. In case the suffix ends with _, the last character is omitted.
+    @param values: 
+        metric values to be logged supposed to be auc_roc, roc, accuracy, sensitivity and specificity in this order.
+    @param plot_tpr_fpr_path: 
+        Path for figure to save the plotted tpr against fpr. In case it is None, the figure will not be created.
+    return 
+        A dictionary containing values from auc computation as part of the evaluation. Returned values include
+        auc value, false positive rate, true positive rate an thresholds. 
+        Dictionary keys are ['auc', 'fpr', 'tpr' and 'thresholds'] respectively.
+    '''
+    wandb_suffix = suffix[:-1] if suffix.endswith('_') else suffix
+    auc_roc, roc, acc, sens, spec = values
+    wandb_dict = {}
+    results = {}
+    auc = -1
+    for name, value in [("Accuracy", acc[suffix]), ("AUC_ROC", auc_roc[suffix]),
+                        ("sensitivity", sens[suffix]), ("specificity", spec[suffix])]:
+        print(name + " " + value)
+        wandb_dict[f"{name}/{wandb_suffix}"] = value.compute()
+        if name == "AUC_ROC":
+            results['auc'] = wandb_dict[f"{name}/{wandb_suffix}"]
+        value.reset()
+
+    if plot_tpr_fpr_path:
+        fpr, tpr, thresholds = roc[suffix].compute()
+        results["fpr"] = fpr
+        results["tpr"] = tpr
+        results["thresholds"] = thresholds
+        results = plot_auc(results, plot_tpr_fpr_path, suffix)
+
+    wandb.log(wandb_dict)
+    return results
+
+
+def eval_epoch_subjects(model, criterion, data_loader, device, values, suffix: str, plot_tpr_fpr=None, scaler=None,
+                        title_suffix="_"):
+    """
+    Evaluation loop for one epoch when using subjects loader; handles subjects that contain data itself, label and path.
+    :param model
+        the model to evaluate
+    :criterion
+        criterion with which to evaluate
+    :data_loader
+        Subjects loader providing the subjects containing data, label (which can be mask or integer) and path.
+    :device
+        the device on which data shall be evaluated
+    :values
+        Module dictionaries for each metric that shall be evaluated. Should contain auc, roc, accuracy, sensitivity, specificity in the given order.
+    :param suffix 
+        used for printouts and also key to update evaluation metrices as roc, auc, and others.
+    :param plot_tpr_fpr
+        Indicates weither to plot roc curve in case this parameter is not None. If it is None, no figure will be created. 
+        Otherwise this parameter should contain a string defining the path to the file as which the plot will be saved.
+    :param scaler 
+        scaler to be used otherwise, None
+    :title_suffix
+        string for printout to indicate weither this is an evaluation or test epoch.
+    """
+    start = time.time()
+    model.eval()
+    torch.set_grad_enabled(False)
+    epoch_loss = 0
+    wandb_suffix = suffix[:-1] if suffix.endswith('_') else suffix
+
+    set_length = len(data_loader)
+    batch_size = data_loader.batch_size or 1
+    y_pred = torch.zeros((set_length, batch_size), dtype=torch.int)  # predicted labels
+    y_true = torch.zeros((set_length, batch_size), dtype=torch.int)  # true labels
+    y_probs = torch.zeros((set_length, batch_size), dtype=torch.float)  # predicted probabilities for cancer
+    for batch_id, batch in enumerate(data_loader):
+        data = batch['image']['data']
+        ground_truth = batch['label']
+        data = data.to(device)
+        if not torch.is_tensor(ground_truth):
+            ground_truth = torch.tensor(ground_truth)
+        ground_truth = ground_truth.to(device)
+
+        if scaler is not None:
+            with torch.autocast(device_type=device, dtype=torch.float16):
+                pred_probs = model(data.float())
+                loss = criterion(pred_probs[:, 0].float(), ground_truth.float())
+        else:
+            pred_probs = model(data.float())
+            loss = criterion(pred_probs[:, 0].float(), ground_truth.float())
+
+        prediction = pred_probs[:, 0] > 0
+        # store prediction results for ROC
+        y_true[batch_id][0:len(ground_truth)] = ground_truth
+        y_pred[batch_id][0:len(ground_truth)] = prediction
+        y_probs[batch_id][0:len(ground_truth)] = pred_probs[:, 0]
+
+        if values is not None:
+            update_evaluation_metrics(suffix, pred_probs, ground_truth, values)
+
+        wandb.log({"loss/" + wandb_suffix: loss.item()}, commit=False)
+        epoch_loss += loss.item()
+
+    avg_epoch_loss = epoch_loss / len(data_loader)
+    end = time.time()
+    print('Validation ({}) {}: Duration = {} sec\n\tAverage Loss: {:.6f}'.format(
+        suffix,
+        title_suffix,
+        end - start,
+        avg_epoch_loss))
+
+    results = {"y_pred": y_pred, "y_prob": y_probs, "y_true": y_true}
+    if values is not None:
+        auc_results = log_evaluation(suffix, values, plot_tpr_fpr)
+        for key, value in auc_results:
+            results[
+                key] = value  #"nn_auc": auc_results["auc"], "fpr": auc_results["fpr"], "tpr": auc_results["tpr"], "thesholds": auc_results["thresholds"]}
+    return avg_epoch_loss, results
+
+
+def get_evaluation_metrics(device=CPU):
+    auc_roc = nn.ModuleDict({state: AUROC(**{"task": "binary"}).to(device) for state in ["train_", "eval_", "Test_"]})
+    roc = nn.ModuleDict({state: ROC(**{"task": "binary"}).to(device) for state in ["train_", "eval_", "Test_"]})
+    acc = nn.ModuleDict({state: Accuracy(**{"task": "binary"}).to(device) for state in ["train_", "eval_", "Test_"]})
+    sens = nn.ModuleDict({state: Precision(**{"task": "binary"}).to(device) for state in ["train_", "eval_", "Test_"]})
+    spec = nn.ModuleDict(
+        {state: Specificity(**{"task": "binary"}).to(device) for state in ["train_", "eval_", "Test_"]})
+    values = (auc_roc, roc, acc, sens, spec)
+    return values
+
+

auto_detect_breast_mri/training/lr_scheduler.py

@@ -0,0 +1,65 @@
+import math
+
+from torch.optim.lr_scheduler import StepLR, LinearLR, MultiStepLR, CosineAnnealingLR, ReduceLROnPlateau
+
+
+class LRScheduler_Manager(object):
+    def __init__(self, optimizer, num_of_epochs):
+        if optimizer is not None:
+            self.optimizer = optimizer
+        else:
+            raise ValueError("LRScheduler_Manager requires an optimizer. Can not be None")
+
+        self.num_of_epochs = num_of_epochs
+
+    def get_scheduler(self, scheduler_name: str):
+        print("get scheduler " + scheduler_name)
+        if scheduler_name == "StepLR":
+            return self.get_stepLR_scheduler()
+        elif scheduler_name == "LinearLR":
+            return self.get_linearLR_scheduler()
+        elif scheduler_name == "MultiStepLR":
+            return self.get_multistepLR_scheduler()
+        elif scheduler_name == "CosineAnnealingLR":
+            return self.get_cosineAnnealingLR_scheduler()
+        elif scheduler_name == "ReduceLROnPlateau":
+            return self.get_ReduceLROnPlateau_scheduler()
+
+    def get_stepLR_scheduler(self):
+        step_size = 10
+        gamma = 0.7
+        step_scheduler = StepLR(self.optimizer, step_size=step_size, gamma=gamma)
+        parameters = "StepLR: step_size=" + str(step_size) + ", gamma=" + str(gamma)
+        return step_scheduler, parameters
+
+    def get_linearLR_scheduler(self):
+        start_factor = 1.0
+        end_factor = 0.000001
+        scheduler = LinearLR(self.optimizer, start_factor=start_factor, end_factor=end_factor,
+                             total_iters=self.num_of_epochs)
+        parameters = "LinearLR: start_factor=" + str(start_factor) + ", end_factor=" + str(
+            end_factor) + ", total_iters=" + str(self.num_of_epochs)
+        return scheduler, parameters
+
+    def get_multistepLR_scheduler(self):
+        milestones = [math.floor(self.num_of_epochs * 0.3), math.floor(self.num_of_epochs * 0.4),
+                      math.floor(self.num_of_epochs * 0.6),
+                      math.floor(self.num_of_epochs * 0.75),
+                      math.floor(self.num_of_epochs * 0.9),
+                      math.floor(self.num_of_epochs * 0.95), math.floor(self.num_of_epochs * 0.975),
+                      math.floor(self.num_of_epochs * 0.99)]
+        gamma = 0.7
+        scheduler = MultiStepLR(self.optimizer, milestones=milestones, gamma=gamma)
+        parameters = "MultiStepLR: milestones=" + str(milestones) + ", gamma=" + str(gamma)
+        return scheduler, parameters
+
+    def get_cosineAnnealingLR_scheduler(self):
+        scheduler = CosineAnnealingLR(self.optimizer, T_max=self.num_of_epochs)
+        return scheduler, ""
+    
+    def get_ReduceLROnPlateau_scheduler(self):
+        factor = 0.1
+        patience = 5
+        scheduler = ReduceLROnPlateau(self.optimizer, mode='min', factor=factor, patience=patience)
+        return scheduler, f"ReduceLROnPlateau: mode={str(scheduler.mode)}, factor={factor}, patience={patience}"
+    

config.example.yaml

@@ -0,0 +1,57 @@
+# Site configuration for auto_detect_breast_mri.
+#
+#   cp config.example.yaml config.yaml      # config*.yaml is gitignored, this example is not
+#   export AIMRI_CONFIG=/absolute/path/to/config.yaml
+#
+# Every value is optional: command line arguments always take precedence, so a fully specified
+# command line needs no config file at all. Keep the real file out of the repository -- it is the
+# one place where paths of your own machine or cluster are allowed to appear.
+
+# --- paths -------------------------------------------------------------------------------------
+# Root folder of the NIfTI data, i.e. the folder holding one subfolder per examination.
+# See README.md ("Data contract") for the two supported folder layouts.
+data_root: /path/to/nifti_root
+
+# Metadata export (.csv or .xlsx) with one row per examination and the label columns configured
+# under `columns:` below.
+metadata_file: /path/to/metadata.csv
+
+# Folder holding the cross validation split files, i.e. the parent of fold0/, fold1/, ...
+# Expected file names are documented in README.md ("Split files").
+split_root: /path/to/split_files
+
+# Where runs write checkpoints, ROC plots, heatmaps, prediction tables and pickled results.
+output_root: /path/to/outputs
+
+# Where already trained checkpoints are read from. Defaults to output_root when unset.
+model_root: /path/to/checkpoints
+
+# --- weights & biases --------------------------------------------------------------------------
+# Project and (optional) team the runs are logged to. Set wandb_mode to 'offline' to keep runs
+# local, or 'disabled' to turn logging off entirely -- useful when running on data that must not
+# leave the machine, since run names and captions can contain case identifiers.
+wandb_project: my-breast-mri-project
+wandb_entity: my-team
+wandb_mode: online
+
+# --- CSV column names --------------------------------------------------------------------------
+# Names of the columns in your metadata export and split files. Override what differs in your data. Values in the side
+# column are matched case insensitively. Unset keys fall back to the defaults in
+# auto_detect_breast_mri/config.py (DEFAULT_COLUMNS), which are the original UKA export's names.
+#
+# tools/make_synthetic_dataset.py reads this block to name the columns of the CSVs it writes and
+# copies it into the config it generates, so a renamed column only has to be changed here.
+columns:
+  examination_id: exam_id           # one MRI examination, in the split CSVs
+  metadata_examination_id: exam_id  # the same ID in the metadata export
+  patient_id: pat_id                          # patient; splits are grouped by this so that no
+                                           # patient appears in two folds
+  side: side                               # breast side in the split CSVs
+  metadata_side: side                     # breast side in the metadata export
+  criterion: Malign                        # training label, one row per breast
+  criterion_left: Malign_Left              # per-side label in the wide metadata export
+  criterion_right: Malign_Right
+  side_left: left                          # the two values found in the side column
+  side_right: right
+  indication_left: Indication_Left         # per-side indication code in the metadata export
+  indication_right: Indication_Right

pyproject.toml

@@ -0,0 +1,25 @@
+[project]
+name = "auto-detect-breast-mri"
+description = "Malignancy classification on breast MRI: abbreviated (AP) against full dynamic (FDP) protocol"
+readme = "README.md"
+requires-python = ">=3.10"
+authors = [{ name = "Debora Jutz" }]   # allow-private-path: package author, published deliberately
+
+# The version stays in auto_detect_breast_mri/__init__.py and the dependency list in
+# requirements.txt, so conda users, CI and uv all read the same two files.
+dynamic = ["version", "dependencies"]
+
+[project.urls]
+Repository = "https://github.com/deboraJ1/auto_detect_breast_mri"  # allow-private-path: public URL
+
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.dynamic]
+version = { attr = "auto_detect_breast_mri.__version__" }
+dependencies = { file = ["requirements.txt"] }
+
+# Only the importable package is shipped; scripts/, jobs/, tools/ and tests/ are repository content.
+[tool.setuptools.packages.find]
+include = ["auto_detect_breast_mri*"]

requirements.txt

@@ -0,0 +1,26 @@
+numpy~=1.26.4
+matplotlib~=3.9.2
+pandas~=2.2.2
+monai
+setuptools~=75.5.0
+torchvision~=0.19.1
+torchio
+wandb~=0.18.0
+scikit-learn~=1.5.2
+scikit-image~=0.24.0
+pillow~=11.0.0
+nibabel
+openpyxl
+
+torch~=2.4.1
+pydicom~=2.4.3
+SimpleITK~=2.4.0
+tqdm~=4.67.0
+scipy~=1.14.1
+sympy~=1.13.2
+tcia_utils~=2.2.1
+imageio~=2.36.0
+opencv-python~=4.10.0.84
+torchmetrics~=1.4.2
+grad-cam~=1.5.4
+PyYAML~=6.0

tests/test_augmentation.py

@@ -0,0 +1,144 @@
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy
+import torch
+import cv2
+
+from auto_detect_breast_mri.data.transforms import *
+from auto_detect_breast_mri.models.unet_crop import UNet_Crop
+from auto_detect_breast_mri.data.loaders import get_multiple_subjects_dataloader
+from auto_detect_breast_mri.config import resolve_path
+from auto_detect_breast_mri.data.metadata import get_uka_metatensor
+import unittest
+
+
+class TestAugmentation(unittest.TestCase):
+    def setUp(self) -> None:
+        # Parameters that can be adapted:
+        self.batch_size = 1
+        # data locations come from the site config; point $AIMRI_CONFIG at the synthetic dataset
+        # (tools/make_synthetic_dataset.py) to run this without any patient data
+        feature_path = resolve_path(None, "metadata_file", "metadata export")
+        path_base = resolve_path(None, "data_root", "root folder of the NIfTI data")
+        split_files_folder = resolve_path(None, "split_root", "folder holding the split files")
+        pretrained_model = str(Path(resolve_path(None, "model_root", "checkpoint folder")) / "unet_best.pth")
+        protocol = ['pre']
+        pre_image_shape = (32, 512, 512)
+
+        # transformations
+        self.norm = ZNormalization()
+        self.crop = UNet_Crop(input_channel=len(protocol), path_to_unet=pretrained_model)
+        self.mirror = tio.RandomFlip((0,1,2), flip_probability=0.5)
+        self.scale = tio.RandomAffine(scales=0.2, degrees=0, center='image', default_pad_value='minimum', check_shape=True)
+        self.rotate = tio.RandomAffine(scales=0, degrees=25, center='image', default_pad_value='minimum', check_shape=True)
+        transform = tio.Compose([
+                self.scale,
+                self.mirror,
+                self.crop,
+                self.norm,
+                ImageToTensor()
+            ])
+
+        # Dataset and DataLoader
+        feature_dataframe = get_uka_metatensor(0, feature_path)
+        self.train_loader, self.eval_loader, self.test_loader = get_multiple_subjects_dataloader(path_base,
+                                                                feature_dataframe,
+                                                                pre_image_shape,
+                                                                protocol=protocol,
+                                                                set_folder=split_files_folder,
+                                                                transform=transform,
+                                                                batch_size=self.batch_size,
+                                                                segmentation_task=True,
+                                                                stratified=True)
+        self.dataset = self.train_loader.dataset
+
+        self.circle_image = numpy.zeros(pre_image_shape)
+        self.circle_center = (112, 357)  # col, row
+        self.radius = 84
+        self.mask_width = 256
+        for d in range(pre_image_shape[0]):
+            half = pre_image_shape[0] / 2
+            radius = self.radius * d / pre_image_shape[0] if d < half else self.radius * (pre_image_shape[0] - d) / \
+                                                                           pre_image_shape[0]
+            radius = int(radius)
+            cv2.circle(self.circle_image[d], self.circle_center, radius, (1, 1, 1), -1)
+
+        self.probability_tensor = torch.ones((2,) + pre_image_shape)
+        self.probability_tensor[1][self.circle_image == 1] = 2.2
+        self.probability_tensor[0][self.circle_image == 1] = 0.1
+        self.probability_tensor[1][self.circle_image != 1] = 0.3
+        self.probability_tensor[0][self.circle_image != 1] = 0.7
+
+        plt.imshow(self.circle_image[16], cmap='gray', interpolation='none')
+        plt.show()
+
+    def test_mask_extraction(self):
+        height_position = self.circle_center[1]  # radius is defined as x, y (x is horizontal)
+        height_min = height_position - int(self.mask_width / 2)
+        height_max = height_position + int(self.mask_width / 2)
+        if height_min < 0:
+            # cropping area would move out of image to the left
+            height_min = 0
+            height_max = self.mask_width
+        elif height_max > self.circle_image.shape[-1]:
+            # cropping area would move out of image to the right
+            height_min = self.mask_width
+            height_max = self.circle_image.shape[-1]
+
+        expected_output = torch.Tensor(self.circle_image)[:, height_min:height_max, :]
+        plt.imshow(expected_output[16], cmap='gray', interpolation='none')
+        plt.show()
+
+        cropped_mask, (comp_min, comp_max) = self.crop.extract_breast_mask(self.probability_tensor, self.mask_width)
+        calculated_output = cropped_mask[0]
+        plt.imshow(cropped_mask[0, 16], cmap='gray', interpolation='none')
+        plt.show()
+
+        self.assertTrue(torch.all(calculated_output.eq(expected_output)))
+
+    def test_augmentation_steps_visually(self):
+        for batch_id, batch in enumerate(self.train_loader):
+            data = batch['image']['data']
+            mask = batch['label']['data']
+            # image (one half)
+            plt.imshow(data[0][0][21], cmap='gray', interpolation='none')
+            plt.show()
+            # mask
+            plt.imshow(mask[0][1][21], cmap='gray', interpolation='none')
+            plt.show()
+
+            # Preprocessing:
+            # 0. norm images_full_data to be between 0 and 1
+            preprocessed_input = self.norm.__call__(data[0])
+            # image (one half)
+            plt.imshow(preprocessed_input[0][21], cmap='gray', interpolation='none')
+            plt.show()
+
+            # 1. crop images_full_data with unet
+            cropped_image, (w_min, w_max), _ = self.crop.__call__(preprocessed_input)
+            cropped_mask = mask[0, :, :, :, w_min:w_max]
+            plt.imshow(torch.cat([cropped_image[0][21], cropped_mask[1][21]]), cmap='gray', interpolation='none')
+            plt.show()
+
+            transformation_tensor = torch.cat([cropped_image, cropped_mask], dim=0)
+            # 2. Mirror randomly
+            flipped_image = self.mirror.__call__(transformation_tensor)
+            plt.imshow(torch.cat([flipped_image[0][21], flipped_image[4][21]]), cmap='gray', interpolation='none')
+            plt.show()
+
+            # 3. Random rotation between [-25, 25]
+            rotated_image = self.rotate.__call__(flipped_image)
+            plt.imshow(torch.cat([rotated_image[0][21], rotated_image[4][21]]), cmap='gray', interpolation='none')
+            plt.show()
+
+            # 4. Rescale between [0.9, 1.1]
+            rescaled_image = self.scale.__call__(rotated_image)
+            plt.imshow(torch.cat([rescaled_image[0][21], rescaled_image[4][21]]), cmap='gray', interpolation='none')
+            plt.show()
+
+        self.assertEqual(batch_id, len(self.train_loader)/self.batch_size)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_dim_comparison.py

@@ -0,0 +1,87 @@
+import unittest
+import torch
+from auto_detect_breast_mri.data.dim_comparison import *
+class TestDimComparison(unittest.TestCase):
+    def setUp(self) -> None:
+        # prepare input tuples to be compared
+        ## tuple 1
+        self.dim1 = torch.randint(1, 5, (1,))
+
+        self.shape_short = (torch.randint(1, 512, (1,)),)
+        while len(self.shape_short) < self.dim1:
+            self.shape_short = (self.shape_short + (torch.randint(1, 512, (1,)),))
+
+        ## tuple 2: longer shape and having value being greater in the longer part
+        self.dim2 = torch.randint(self.dim1, 10, (1,))
+        self.shape_long_greater = (self.shape_short + (700,))  # 700 as value being for sure greater than max value 512
+
+        ## tuple 3: longer shape and having value being smaller in the longer part
+        self.shape_long_smaller = (self.shape_short + (0,))
+
+        ## tuple 4: same length but all values smaller
+        self.shape_short_smaller = (self.shape_short[0] - 1,)
+        length = len(self.shape_short_smaller)
+        while length < self.dim1:
+            self.shape_short_smaller = (self.shape_short_smaller + (self.shape_short[length] - 1,))
+            length += 1
+
+        ## tuple 5: same length but all values greater
+        self.shape_short_greater = (self.shape_short[0] + 1,)
+        length = len(self.shape_short_greater)
+        while length < self.dim1:
+            self.shape_short_greater = (self.shape_short_greater + (self.shape_short[length] + 1,))
+            length += 1
+
+    def test_is_any_dim_greater_allSmaller(self):
+        # compare with tuple with same size, all values being smaller
+        self.assertEqual(is_any_dim_greater(self.shape_short, self.shape_short_smaller), True)
+
+    def test_is_any_dim_greater_allEqual(self):
+        self.assertEqual(is_any_dim_greater(self.shape_short, self.shape_short), False)
+
+    def test_is_any_dim_greater_SmallerLong(self):
+        # compare with tuple with longer size, values being out of bound being smaller
+        self.assertEqual(is_any_dim_greater(self.shape_short, self.shape_long_smaller), False)
+
+    def test_is_any_dim_greater_GreaterLong(self):
+        # compare with tuple with smaller size, value out of bounds being greater
+        self.assertEqual(is_any_dim_greater(self.shape_long_greater, self.shape_short), False)
+
+    def test_is_any_dim_greater_GreaterLong2(self):
+        # compare with tuple with longer size, out of dimensions value being greater
+        self.assertEqual(is_any_dim_greater(self.shape_short, self.shape_long_greater), False)
+
+    def test_is_any_dim_greater_Greater(self):
+        # compare with tuple with same size, single value being greater
+        random_pos = torch.randint(0, self.dim1 - 1)
+        greater = (self.shape_short[:random_pos] + (self.shape_short[random_pos] + 1,) + self.shape_short[random_pos:])
+        self.assertEqual(is_any_dim_greater(greater, self.shape_short), True)
+
+    def test_is_any_dim_smaller_allGreater(self):
+        # all positions being smaller
+        self.assertEqual(is_any_dim_smaller(self.shape_short, self.shape_short_greater), True)
+
+    def test_is_any_dim_smaller_allEqual(self):
+        self.assertEqual(is_any_dim_smaller(self.shape_short, self.shape_short), False)
+
+    def test_is_any_dim_smaller_SmallerLong(self):
+        # compare with tuple with same size, all values being greater
+        self.assertEqual(is_any_dim_smaller(self.shape_short, self.shape_short_smaller), False)
+
+    def test_is_any_dim_smaller_SmallerLong2(self):
+        # compare with tuple with same size, all values being smaller
+        self.assertEqual(is_any_dim_smaller(self.shape_short_smaller, self.shape_short), True)
+
+    def test_is_any_dim_smaller_GreaterLong2(self):
+        self.assertEqual(is_any_dim_smaller(self.shape_short, self.shape_long_greater), False)
+
+    def test_is_any_dim_smaller_Smaller(self):
+        # single position has smaller value
+        random_pos = torch.randint(0, self.dim1 - 1)
+        smaller = (self.shape_short[:random_pos] + (self.shape_short[random_pos] - 1,) + self.shape_short[random_pos:])
+
+        self.assertEqual(is_any_dim_smaller(smaller, self.shape_short), True)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_training.py

@@ -0,0 +1,7 @@
+"""
+Placeholder for unit tests of the training and evaluation loops in
+auto_detect_breast_mri.training.loops.
+
+TODO: test one training epoch and one evaluation epoch against the synthetic dataset
+(tools/make_synthetic_dataset.py), so the loops are covered without any patient data.
+"""

tools/check_no_private_paths.py

@@ -0,0 +1,202 @@
+#!/usr/bin/env python3
+"""
+Fail if a tracked file contains something that must not be published: an absolute path of somebody's
+machine, a cluster account, a personal name, or a bare identifier that could be a patient or
+examination ID.
+
+    python tools/check_no_private_paths.py            # scan the whole repository
+    python tools/check_no_private_paths.py a.py b.py  # scan the given files (pre-commit passes these)
+
+Exit status is 1 if anything was found, so this works as a pre-commit hook and as a CI step.
+
+Two escape hatches, both deliberate and both visible in review:
+  * put `# allow-private-path` (or `allow-private-path` in any comment syntax) on the offending line
+  * add the file to ALLOWED_FILES below
+
+Digits are only reported inside string literals of Python files and in plain text elsewhere, so
+seeds, random states and version numbers written as integers do not trip the check.
+"""
+
+import argparse
+import ast
+import re
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+
+# Directories that are never scanned.
+SKIP_DIRS = {".git", "__pycache__", ".venv", "venv", "env", "build", "dist", ".pytest_cache",
+             ".mypy_cache", ".ruff_cache", ".idea", ".vscode", "outputs", "wandb", "synthetic_data"}
+
+# Only these extensions are scanned; everything else (images, checkpoints, data) is either
+# gitignored or not text.
+SCAN_SUFFIXES = {".py", ".sh", ".sbatch", ".example", ".yaml", ".yml", ".cfg", ".ini", ".toml",
+                 ".md", ".txt", ".env", ".json", ".zsh", ".bash"}
+
+# Files exempt from the whole check. Keep this list empty if you can.
+ALLOWED_FILES = {
+    "tools/check_no_private_paths.py",   # this file necessarily contains the patterns themselves
+}
+
+# The untracked site configuration: these files exist precisely to hold local paths, and .gitignore
+# keeps them out of the repository. Mirrors the first section of .gitignore -- keep both in sync.
+SKIP_GLOBS = ["config*.yaml", "config*.yml", "jobs/site.env", "*.sbatch", ".env"]
+SKIP_GLOB_EXCEPTIONS = ["config.example.yaml", "*.sbatch.example", "*.env.example"]
+
+ESCAPE_MARKER = "allow-private-path"
+
+# Personal names that must not appear anywhere. Extend when people join or leave the project.
+NAME_DENYLIST = ["debora", "gustav", "it336446"]
+
+# (name, compiled pattern, explanation) applied line by line to every scanned file.
+LINE_PATTERNS = [
+    ("unix-home", re.compile(r"/home/[A-Za-z0-9._-]+"),
+     "absolute path in somebody's home directory"),
+    ("hpc-scratch", re.compile(r"/(?:hpcwork|work|rwthfs|mnt|media)/[A-Za-z0-9._/-]+"),
+     "absolute path on a cluster or mounted filesystem"),
+    ("windows-drive", re.compile(r"\b[A-Za-z]:[\\/]{1,2}[A-Za-z0-9._\\/-]+"),
+     "absolute Windows path"),
+    ("cluster-account", re.compile(r"\bp00\d{5}\b"),
+     "cluster accounting project"),
+    ("cluster-project", re.compile(r"\brwth\d{3,}\b", re.IGNORECASE),
+     "cluster project id"),
+    ("personal-name", re.compile("|".join(re.escape(name) for name in NAME_DENYLIST), re.IGNORECASE),
+     "personal name or user account"),
+]
+
+# Bare 7 to 10 digit numbers look like examination or patient IDs. Applied to Python string
+# literals and to non-Python text, never to Python integer literals.
+ID_PATTERN = re.compile(r"(?<!\d)\d{7,10}(?!\d)")
+ID_REASON = "bare 7-10 digit number, which may be a patient or examination identifier"
+
+# Digits inside a URL are page ids, dataset ids or versions, never a local patient identifier.
+URL_PATTERN = re.compile(r"https?://")
+
+
+class Finding:
+    def __init__(self, path, line_no, rule, reason, excerpt):
+        self.path, self.line_no, self.rule, self.reason, self.excerpt = path, line_no, rule, reason, excerpt
+
+    def __str__(self):
+        return f"{self.path}:{self.line_no}: [{self.rule}] {self.reason}\n      {self.excerpt.strip()[:160]}"
+
+
+def is_untracked_site_file(path: Path) -> bool:
+    """True for the gitignored files whose whole purpose is to hold local paths."""
+    if any(path.match(pattern) for pattern in SKIP_GLOB_EXCEPTIONS):
+        return False
+    return any(path.match(pattern) or path.name == pattern for pattern in SKIP_GLOBS)
+
+
+def is_scannable(path: Path) -> bool:
+    if any(part in SKIP_DIRS for part in path.parts):
+        return False
+    if is_untracked_site_file(path):
+        return False
+    if path.name.startswith("."):
+        # dotfiles that are still worth scanning
+        return path.name in {".gitignore", ".pre-commit-config.yaml"}
+    return path.suffix in SCAN_SUFFIXES
+
+
+def iter_files(explicit):
+    if explicit:
+        for name in explicit:
+            path = Path(name)
+            if path.is_file() and is_scannable(path):
+                yield path
+        return
+    for path in sorted(REPO_ROOT.rglob("*")):
+        if path.is_file() and is_scannable(path.relative_to(REPO_ROOT)):
+            yield path.relative_to(REPO_ROOT)
+
+
+def string_literal_lines(source: str):
+    """
+    Python string literals as (first line, last line, value), for the ID check.
+
+    The line span matters for two reasons: a multi-line literal is reported at its first line, and
+    an escape marker anywhere inside it (or on its first line) must silence it.
+    """
+    try:
+        tree = ast.parse(source)
+    except SyntaxError:
+        return []
+    found = []
+    for node in ast.walk(tree):
+        if isinstance(node, ast.Constant) and isinstance(node.value, str):
+            found.append((node.lineno, getattr(node, "end_lineno", node.lineno) or node.lineno, node.value))
+    return found
+
+
+def scan(path: Path):
+    relative = path.as_posix()
+    if relative in ALLOWED_FILES:
+        return []
+    try:
+        source = (REPO_ROOT / path).read_text(encoding="utf-8", errors="replace")
+    except OSError as exc:
+        print(f"WARNING: cannot read {path}: {exc}", file=sys.stderr)
+        return []
+
+    lines = source.splitlines()
+    findings = []
+    for line_no, line in enumerate(lines, start=1):
+        if ESCAPE_MARKER in line:
+            continue
+        for rule, pattern, reason in LINE_PATTERNS:
+            if pattern.search(line):
+                findings.append(Finding(relative, line_no, rule, reason, line))
+
+    if path.suffix == ".py":
+        for first_line, last_line, value in string_literal_lines(source):
+            span = lines[first_line - 1:last_line]
+            if any(ESCAPE_MARKER in line for line in span):
+                continue
+            for value_line in value.splitlines() or [value]:
+                if URL_PATTERN.search(value_line):
+                    continue
+                if ID_PATTERN.search(value_line):
+                    excerpt = lines[first_line - 1] if first_line <= len(lines) else value_line
+                    findings.append(Finding(relative, first_line, "possible-identifier", ID_REASON,
+                                            value_line if len(span) > 1 else excerpt))
+                    break
+    else:
+        for line_no, line in enumerate(lines, start=1):
+            if ESCAPE_MARKER in line or URL_PATTERN.search(line):
+                continue
+            if ID_PATTERN.search(line):
+                findings.append(Finding(relative, line_no, "possible-identifier", ID_REASON, line))
+
+    return findings
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__,
+                                     formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("files", nargs="*", help="files to scan; default is the whole repository")
+    parser.add_argument("-q", "--quiet", action="store_true", help="only print the summary")
+    args = parser.parse_args()
+
+    findings = []
+    scanned = 0
+    for path in iter_files(args.files):
+        scanned += 1
+        findings.extend(scan(path))
+
+    if findings and not args.quiet:
+        print("Found content that must not be published:\n")
+        for finding in findings:
+            print(f"  {finding}")
+        print()
+        print("Fix by moving the value into your untracked config.yaml / jobs/site.env and")
+        print("reading it via auto_detect_breast_mri.config, or -- if the match is a false positive -- append")
+        print(f"'# {ESCAPE_MARKER}' to that line.")
+
+    print(f"{'FAIL' if findings else 'OK'}: {len(findings)} finding(s) in {scanned} scanned file(s).")
+    return 1 if findings else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/check_split_fractions.py

@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+"""
+Verify the nesting guarantee of the training fraction files: every smaller fraction must be a
+subset of every larger one, for every fold.
+
+    python tools/check_split_fractions.py [--split-root <folder>] [--folds 5]
+
+The split root comes from the site config when it is not given (see config.example.yaml).
+"""
+
+import argparse
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+from auto_detect_breast_mri.config import column, resolve_path
+
+
+def fraction_of(filename: str) -> float:
+    """0.05 from '..._frac0.05.csv'."""
+    return float(filename.split("_frac")[-1].rsplit(".csv", 1)[0])
+
+
+def check_fold(fold_dir: Path, id_column: str, expected_files: int) -> bool:
+    """True if every fraction file in fold_dir is nested in all larger ones."""
+    fraction_files = sorted(fold_dir.glob("*_frac0.*.csv"), key=lambda path: fraction_of(path.name))
+    if len(fraction_files) != expected_files:
+        print(f"WARNING: {fold_dir.name}: expected {expected_files} fraction files, "
+              f"found {len(fraction_files)}: {[path.name for path in fraction_files]}", file=sys.stderr)
+    ids = {fraction_of(path.name): set(pd.read_csv(path)[id_column]) for path in fraction_files}
+
+    ok = True
+    for child, child_ids in sorted(ids.items()):
+        for parent, parent_ids in sorted(ids.items()):
+            if parent <= child:
+                # a larger set can not be expected inside a smaller one; also skips parent == child
+                continue
+            missing = child_ids - parent_ids
+            if missing:
+                ok = False
+                print(f"ERROR: {fold_dir.name}: fraction {child} is NOT contained in {parent}: "
+                      f"{len(missing)} entries missing from the parent.", file=sys.stderr)
+            else:
+                print(f"  {fold_dir.name}: fraction {child} is fully covered by {parent}.")
+    return ok
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__,
+                                     formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--split-root", default=None,
+                        help="folder holding fold<k>/. Config key: split_root")
+    parser.add_argument("--folds", type=int, default=5, help="number of folds to check. Default: 5")
+    parser.add_argument("--expected-files", type=int, default=4,
+                        help="fraction files expected per fold. Default: 4")
+    args = parser.parse_args()
+
+    split_root = Path(resolve_path(args.split_root, "split_root", "folder with the split files"))
+    if not split_root.exists():
+        sys.exit(f"ERROR: directory {split_root} does not exist.")
+
+    id_column = column("examination_id")
+    all_ok = True
+    for fold in range(args.folds):
+        fold_dir = split_root / f"fold{fold}"
+        if not fold_dir.exists():
+            print(f"WARNING: {fold_dir} does not exist, skipping.", file=sys.stderr)
+            continue
+        print(f"Investigating {fold_dir}")
+        all_ok &= check_fold(fold_dir, id_column, args.expected_files)
+
+    print("OK: all fractions are nested." if all_ok else "FAIL: nesting is violated, see above.")
+    sys.exit(0 if all_ok else 1)
+
+
+if __name__ == "__main__":
+    main()

tools/make_synthetic_dataset.py

@@ -0,0 +1,358 @@
+#!/usr/bin/env python3
+"""
+Build a small synthetic dataset so the repository can be run and tested without any patient data.
+
+    python tools/make_synthetic_dataset.py --out synthetic_data
+    export AIMRI_CONFIG=synthetic_data/config.yaml
+    python scripts/train/train.py resnet18_abrv -b 2 -n 1 -c 0 -d 0 -t -e
+
+What is written (see README.md, "Data contract", for the meaning of every part):
+
+    <out>/nifti/                     image root, both supported folder layouts side by side
+        <exam_id>/                     bilateral layout: side is encoded in the file name
+            L-Dyn_0.nii.gz  L-Sub_1.nii.gz  L-Sub_2.nii.gz  L-Sub_3.nii.gz  L-Sub_4.nii.gz  L-T2.nii.gz
+            R-Dyn_0.nii.gz  ...
+            L-mask_breast_nn.nii.gz  R-mask_breast_nn.nii.gz     (only with --masks)
+        <exam_id>_left/                unilateral layout: side is the folder, 'Pre' means Dyn_0
+            Pre.nii.gz  Post_1.nii.gz  Sub_1.nii.gz  Sub_2.nii.gz  Sub_3.nii.gz  Sub_4.nii.gz  T2.nii.gz
+        <exam_id>_right/  ...
+    <out>/metadata.csv               one row per examination, wide (per-side label and indication
+                                     columns)
+    <out>/split_files/fold<k>/       stratified_{training,evaluation,test}_set-f<k>[-<subfold>].csv
+    <out>/config.yaml                a ready to use site config pointing at the above, including
+                                     the `columns:` block describing the generated CSVs
+
+Column names and side values are not defined here: they are read from config.example.yaml, so a
+renamed column has to be changed in that one file only.
+
+Every file holds a single breast, the one its name or folder says: `L-*.nii.gz` and
+`<exam_id>_left/*` are the left breast, `R-*.nii.gz` and `<exam_id>_right/*` the right one. Nothing
+in the pipeline splits a volume into two halves, so a volume showing both breasts would be wrong.
+
+The volumes are a coarse breast phantom: dark air, a bright chest wall slab at the posterior end
+and one bright breast in front of it, plus a small lesion whose intensity correlates with the
+label. The breast sits off the midline towards the patient's midline side, mirrored between left
+and right, so the two sides are not identical volumes. That gives the cropping code the geometry it
+looks for and lets a model overfit the volumes, so a training run produces a non-degenerate ROC.
+They are not images of anything and contain no information about any person.
+"""
+
+import argparse
+import sys
+from pathlib import Path
+
+import numpy as np
+
+# Canonical sequence names of the two layouts. Keep in sync with
+# auto_detect_breast_mri/data/breast_mri_dataset.py (available_protocols / unilateral_mode_mapping).
+BILATERAL_MODES = ["Dyn_0", "Sub_1", "Sub_2", "Sub_3", "Sub_4", "T2"]
+UNILATERAL_FILES = ["Pre", "Post_1", "Sub_1", "Sub_2", "Sub_3", "Sub_4", "T2"]
+MASK_NAME = "mask_breast_nn"
+
+# The `columns:` block of config.example.yaml names the columns of every CSV written here and is
+# copied into the generated config.yaml. It is deliberately the only place those names exist: this
+# generator has no defaults of its own.
+EXAMPLE_CONFIG = Path(__file__).resolve().parent.parent / "config.example.yaml"
+
+# Column keys this generator needs in order to write its CSVs. The example config may list more
+# (metadata_side is only used by the .xlsx metadata modes); all of them are copied to the output.
+REQUIRED_COLUMN_KEYS = ("examination_id", "metadata_examination_id", "patient_id", "side",
+                        "criterion", "criterion_left", "criterion_right",
+                        "side_left", "side_right", "indication_left", "indication_right")
+
+# Indication codes written into the metadata, so the per-indication subgroup analysis has something
+# to group by. Deliberately unbalanced, like a real indication distribution: two large subgroups and
+# a tail of small ones, one of which is too small to bootstrap.
+INDICATION_CODES = (0, 0, 0, 1, 1, 1, 1, 2, 3, 6)
+
+# Sides as this script talks about them internally. The configured values of columns.side_left and
+# columns.side_right are used when writing them to a CSV, the folder/file layout uses these.
+SIDES = ("left", "right")
+
+
+def load_columns(example_config: Path = EXAMPLE_CONFIG) -> dict:
+    """
+    Read the `columns:` block of config.example.yaml, the single definition of the column names.
+
+    The generated config.yaml repeats the block, so the synthetic dataset describes itself and does
+    not depend on DEFAULT_COLUMNS in auto_detect_breast_mri/config.py (which carries the original
+    UKA names and would not match these CSVs).
+    """
+    try:
+        import yaml
+    except ImportError:  # pragma: no cover - depends on the environment
+        sys.exit("ERROR: reading config.example.yaml needs PyYAML: pip install pyyaml")
+    if not example_config.exists():
+        sys.exit(f"ERROR: {example_config} not found; it defines the column names to write.")
+
+    values = yaml.safe_load(example_config.read_text()) or {}
+    columns = values.get("columns") if isinstance(values, dict) else None
+    if not isinstance(columns, dict):
+        sys.exit(f"ERROR: {example_config} has no 'columns:' mapping to take the column names from.")
+
+    missing = [key for key in REQUIRED_COLUMN_KEYS if not columns.get(key)]
+    if missing:
+        sys.exit(f"ERROR: 'columns:' in {example_config} is missing {', '.join(missing)}.")
+    for side in SIDES:
+        value = str(columns[f"side_{side}"])
+        if value.lower() != side:
+            sys.exit(f"ERROR: columns.side_{side} in {example_config} is '{value}', but the side of "
+                     f"a synthetic case is encoded as '{side}' in the folder and file names. The "
+                     f"loaders match the side column against those names case insensitively, so "
+                     f"the value has to lower case to '{side}'.")
+    return {str(key): str(value) for key, value in columns.items() if value}
+
+
+def write_nifti(path: Path, volume: np.ndarray) -> None:
+    """Write one 3D volume as .nii.gz with an identity affine."""
+    try:
+        import nibabel as nib
+    except ImportError:  # pragma: no cover - depends on the environment
+        sys.exit("ERROR: writing NIfTI files needs nibabel (a torchio dependency): pip install nibabel")
+    path.parent.mkdir(parents=True, exist_ok=True)
+    nib.save(nib.Nifti1Image(volume.astype(np.float32), affine=np.eye(4)), str(path))
+
+
+def breast_phantom(shape, side: str = "left"):
+    """
+    Geometry of one synthetic breast, as boolean masks over a (depth, height, width) volume.
+
+    A volume holds the single breast of `side`, exactly as the real data does: the side is carried
+    by the file name (`L-`/`R-`) or the folder name (`<exam_id>_left`), never by one half of a
+    two-breast volume. The axes follow the data contract of the pipeline, so a phantom is croppable
+    the same way a real examination is:
+        axis 0 (depth)   slices
+        axis 1 (height)  anterior (index 0, breast) to posterior (last index, chest wall);
+                         this is the axis UNet_Crop crops along
+        axis 2 (width)   the two sides are mirrored through the middle of this axis, so the left
+                         and right volume of an examination are not the same array
+
+    :return: (body, breast, lesion) boolean arrays; `breast` is the half ellipsoid the breast mask
+        holds and the lesion sits inside it.
+    """
+    depth, height, width = shape
+    z, y, x = np.ogrid[:depth, :height, :width]
+
+    # chest wall: a slab filling the posterior part of the volume
+    chest = y >= int(0.55 * height)
+
+    # one half ellipsoid, centred on the chest wall and reaching into the anterior air. It is
+    # off centre towards the patient's midline, and that offset is mirrored between the sides.
+    breast_y = 0.55 * height
+    breast_radius_y = 0.45 * height
+    breast_radius_x = 0.30 * width
+    breast_radius_z = 0.40 * depth
+    centre_z = (depth - 1) / 2
+    centre_x = 0.40 * width if side == "left" else 0.60 * width
+    ellipsoid = (((z - centre_z) / breast_radius_z) ** 2
+                 + ((y - breast_y) / breast_radius_y) ** 2
+                 + ((x - centre_x) / breast_radius_x) ** 2) <= 1.0
+    breast = ellipsoid & (y <= breast_y)  # anterior half only, the rest is chest wall
+
+    # a small lesion inside the breast
+    lesion_radius = max(2.0, 0.05 * width)
+    lesion = (((z - centre_z) / max(1.0, 0.5 * lesion_radius)) ** 2
+              + ((y - 0.35 * height) / lesion_radius) ** 2
+              + ((x - centre_x) / lesion_radius) ** 2) <= 1.0
+    return chest | breast, breast, lesion
+
+
+def make_volume(rng: np.random.Generator, shape, label: int, side: str = "left",
+                blob: bool = True) -> np.ndarray:
+    """
+    A coarse breast MRI phantom: dark air, a bright chest wall slab and the breast of `side`.
+
+    It is not an image of anything, it only has the large scale geometry the cropping code looks
+    for: air anterior of the breast to crop away, and the single breast the file stands for. The
+    lesion inside it is brighter for a positive label, so a model can still overfit the volumes.
+    """
+    body, breast, lesion = breast_phantom(shape, side)
+    volume = rng.normal(loc=10.0, scale=3.0, size=shape)  # air
+    volume[body] += 90.0                                  # chest wall and breast
+    volume[breast] += 30.0                                # breast a little brighter than the chest
+    if blob:
+        volume[lesion] += 60.0 if label else 15.0
+    return volume
+
+
+def make_mask(shape, side: str = "left") -> np.ndarray:
+    """The breast of `side` as a binary mask (1 inside, 0 outside)."""
+    _, breast, _ = breast_phantom(shape, side)
+    return breast.astype(np.float32)
+
+
+def build_cases(n_examinations: int, first_id: int, n_unilateral: int):
+    """
+    One entry per breast: (examination_id, patient_id, side, label, layout), side one of SIDES.
+
+    Two examinations share a patient ID, so the patient-grouped split has something to group.
+    """
+    cases = []
+    for index in range(n_examinations):
+        examination_id = str(first_id + index)
+        patient_id = str(first_id + index // 2)  # two examinations per patient
+        layout = "unilateral" if index >= n_examinations - n_unilateral else "bilateral"
+        for side_index, side in enumerate(SIDES):
+            # deterministic, roughly balanced labels
+            label = (index + side_index) % 2
+            cases.append((examination_id, patient_id, side, label, layout))
+    return cases
+
+
+def write_images(out_root: Path, cases, shape, with_masks: bool, seed: int) -> None:
+    rng = np.random.default_rng(seed)
+    nifti_root = out_root / "nifti"
+    for examination_id, _, side, label, layout in cases:
+        if layout == "bilateral":
+            prefix = "L" if side == "left" else "R"
+            folder = nifti_root / examination_id
+            for mode in BILATERAL_MODES:
+                write_nifti(folder / f"{prefix}-{mode}.nii.gz", make_volume(rng, shape, label, side))
+            if with_masks:
+                write_nifti(folder / f"{prefix}-{MASK_NAME}.nii.gz", make_mask(shape, side))
+        else:
+            folder = nifti_root / f"{examination_id}_{side}"
+            for name in UNILATERAL_FILES:
+                write_nifti(folder / f"{name}.nii.gz", make_volume(rng, shape, label, side))
+            if with_masks:
+                write_nifti(folder / f"{MASK_NAME}.nii.gz", make_mask(shape, side))
+
+
+def write_metadata(out_root: Path, cases, columns: dict) -> Path:
+    """Wide metadata: one row per examination with a per-side label column."""
+    import pandas as pd
+
+    criterion_of_side = {side: columns[f"criterion_{side}"] for side in SIDES}
+    indication_of_side = {side: columns[f"indication_{side}"] for side in SIDES}
+    rows = {}
+    for index, (examination_id, patient_id, side, label, _) in enumerate(cases):
+        row = rows.setdefault(examination_id, {columns["metadata_examination_id"]: examination_id,
+                                               columns["examination_id"]: examination_id,
+                                               columns["patient_id"]: patient_id})
+        row[criterion_of_side[side]] = label
+        # deterministic, so a regenerated dataset gives the same subgroups
+        row[indication_of_side[side]] = float(INDICATION_CODES[index % len(INDICATION_CODES)])
+    path = out_root / "metadata.csv"
+    pd.DataFrame(list(rows.values())).to_csv(path, index=False)
+    return path
+
+
+def write_splits(out_root: Path, cases, columns: dict, n_folds: int, subfold: int) -> Path:
+    """
+    Long split files, one row per breast, in the layout the loaders build their file names from:
+        <split_root>/fold<k>/stratified_training_set-f<k>-<subfold>.csv
+        <split_root>/fold<k>/stratified_evaluation_set-f<k>-<subfold>.csv
+        <split_root>/fold<k>/stratified_test_set-f<k>.csv
+    Split by patient, so no patient appears in two of the three files of a fold.
+    """
+    import pandas as pd
+
+    split_root = out_root / "split_files"
+    # the examination ID has to stay the first column: the loaders rename it by position
+    frame = pd.DataFrame([{columns["examination_id"]: examination_id,
+                           columns["patient_id"]: patient_id,
+                           columns["side"]: columns[f"side_{side}"],
+                           columns["criterion"]: label}
+                          for examination_id, patient_id, side, label, _ in cases])
+
+    patients = sorted(frame[columns["patient_id"]].unique())
+    # Every patient is the test patient of exactly one fold: the folds take consecutive blocks, and
+    # the leftover of an uneven division is spread over the first folds. Rotating by a single
+    # patient instead would make the test sets of neighbouring folds overlap, which breaks the
+    # out-of-fold analyses (a woman would sit in several test folds).
+    block, leftover = divmod(len(patients), n_folds)
+    bounds = [0]
+    for fold in range(n_folds):
+        bounds.append(bounds[-1] + block + (1 if fold < leftover else 0))
+    for fold in range(n_folds):
+        test_patients = patients[bounds[fold]:bounds[fold + 1]]
+        rest = [patient for patient in patients if patient not in set(test_patients)]
+        n_eval = max(1, len(rest) // 4)
+        eval_patients, train_patients = rest[:n_eval], rest[n_eval:]
+
+        fold_dir = split_root / f"fold{fold}"
+        fold_dir.mkdir(parents=True, exist_ok=True)
+        for name, selected in (("training_set", train_patients),
+                               ("evaluation_set", eval_patients),
+                               ("test_set", test_patients)):
+            suffix = f"-f{fold}" if name == "test_set" else f"-f{fold}-{subfold}"
+            part = frame[frame[columns["patient_id"]].isin(selected)]
+            part.to_csv(fold_dir / f"stratified_{name}{suffix}.csv", index=False)
+    return split_root
+
+
+def write_config(out_root: Path, metadata_file: Path, split_root: Path, columns: dict) -> Path:
+    """
+    A site config pointing at the generated data, usable as $AIMRI_CONFIG.
+
+    The `columns:` block is the one read from config.example.yaml, so the config names exactly the
+    columns that were written, whatever they are called there.
+    """
+    path = out_root / "config.yaml"
+    lines = [
+        "# Generated by tools/make_synthetic_dataset.py -- synthetic data, safe to share.",
+        f"data_root: {(out_root / 'nifti').as_posix()}",
+        f"metadata_file: {metadata_file.as_posix()}",
+        f"split_root: {split_root.as_posix()}/",
+        f"output_root: {(out_root / 'outputs').as_posix()}",
+        f"model_root: {(out_root / 'outputs').as_posix()}",
+        "wandb_mode: disabled",
+        "",
+        "# Column names of the CSVs above, copied from config.example.yaml.",
+        "columns:",
+    ]
+    lines += [f"  {key}: {value}" for key, value in columns.items()]
+    path.write_text("\n".join(lines) + "\n")
+    (out_root / "outputs").mkdir(parents=True, exist_ok=True)
+    return path
+
+
+def build_parser():
+    parser = argparse.ArgumentParser(description=__doc__,
+                                     formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--out", default="synthetic_data", type=Path,
+                        help="output folder. Default: synthetic_data (gitignored)")
+    parser.add_argument("--examinations", type=int, default=8,
+                        help="number of examinations; each yields two breasts. Default: 8")
+    parser.add_argument("--unilateral", type=int, default=4,
+                        help="how many of them use the unilateral folder layout. Default: 4")
+    parser.add_argument("--shape", type=int, nargs=3, default=(32, 512, 512),
+                        metavar=("DEPTH", "HEIGHT", "WIDTH"),
+                        help="volume shape. Default: 16 64 64 (small on purpose)")
+    parser.add_argument("--folds", type=int, default=2, help="number of outer folds. Default: 2")
+    parser.add_argument("--subfold", type=int, default=0, help="inner fold index. Default: 0")
+    parser.add_argument("--first-id", type=int, default=1000001,   # allow-private-path
+                        help="first synthetic examination ID. Default: %(default)s")
+    parser.add_argument("--example-config", type=Path, default=EXAMPLE_CONFIG,
+                        help="config file to take the `columns:` block from. Default: %(default)s")
+    parser.add_argument("--masks", action="store_true",
+                        help="also write breast masks, needed for the segmentation/cropping code")
+    parser.add_argument("--seed", type=int, default=0)
+    return parser
+
+
+def main():
+    args = build_parser().parse_args()
+    if args.unilateral > args.examinations:
+        sys.exit("ERROR: --unilateral cannot exceed --examinations.")
+
+    out_root = args.out.resolve()
+    columns = load_columns(args.example_config)
+    cases = build_cases(args.examinations, args.first_id, args.unilateral)
+
+    print(f"Writing {len(cases)} breasts of {args.examinations} examinations to {out_root}")
+    print(f"  column names from {args.example_config}")
+    write_images(out_root, cases, tuple(args.shape), args.masks, args.seed)
+    metadata_file = write_metadata(out_root, cases, columns)
+    split_root = write_splits(out_root, cases, columns, args.folds, args.subfold)
+    config_file = write_config(out_root, metadata_file, split_root, columns)
+
+    print(f"  images   {out_root / 'nifti'}")
+    print(f"  metadata {metadata_file}")
+    print(f"  splits   {split_root}")
+    print(f"  config   {config_file}")
+    print(f"\nUse it with:\n  export AIMRI_CONFIG={config_file}")
+
+
+if __name__ == "__main__":
+    main()