Interpretable Similarity of Synthetic Image Utility (IUS)
This repository currently contains the official PyTorch implementation of the IUS measure, introduced in "Interpretable Similarity of Synthetic Image Utility" , published in IEEE Transactions on Medical Imaging (TMI).
Status: This repository currently contains the official PyTorch code for IUS.
Pretrained checkpoints are not included yet and will be released separately.
Overview
IUS (Interpretable Utility Similarity) is an interpretable measure for assessing the utility of synthetic medical image datasets for downstream clinical decision support (CDS) tasks, built upon the EPU-CNN framework introduced in E pluribus unum interpretable convolutional neural networks and available in our previous code repository.
Table of Contents
Features
- YAML-based configuration for reproducible training, inference, and evaluation
- Flexible support for multiple training dataset structures
- Integrated TensorBoard logging for experiment monitoring
- Complete implementation of the IUS pipeline:
- EPU-CNN training and classification performance evaluation
- Baseline feature contribution profile estimation
- Synthetic dataset or singleton evaluation using IUS
- Support for both color and grayscale medical imaging modalities
- Reproducible experiment management through structured saving of checkpoints, logs, timestamped results, and evaluation outputs
Installation
Clone the repository:
git clone https://github.com/innoisys/ius.git
cd ius
Install dependencies:
pip install -r requirements.txt
Project Structure
ius/
├── configs/ # YAML configuration files
├── data/ # Data loading and preprocessing
│ ├── data_utils.py # Common utilities for image/data handling
│ ├── dataloader.py # Dataloader definitions
│ ├── dataset.py # Dataset implementations
│ ├── loading.py # Dataset setup from YAML configuration
│ ├── parsers.py # Data parser implementations
│ └── perceptual_transforms.py # PFM (Perceptual Feature Map) generation
├── datasets/ # Real datasets for EPU-CNN training & baseline feature contribution profile estimation
├── datasets_synthetic/ # Synthetic data for IUS evaluation
├── ius/ # IUS implementation
│ ├── ius.py # IUS measure class
│ └── ius_eval_parser.py # Suggested synthetic data parser (not requiring label information)
├── model/ # EPU-CNN model implementation
│ ├── epu.py # Main EPU-CNN model definition
│ ├── module_mapping.py # Mappings from YAML config names to torch.nn layers/activations
│ ├── register_modules.py # Registry for configurable model components
│ ├── subnetworks.py # Subnetwork implementation
│ └── subnetwork_utilities.py # Subnetwork helper modules
├── results/ # Training and inference outputs
│ ├── cb_vectors.py # Saved baseline feature contribution profiles (from infer_cb_vector.py)
│ ├── checkpoints.py # Saved EPU-CNN checkpoints and training configurations (from train_epu.py)
│ ├── classification_performance.py # Classification performance results (from infer_epu.py)
│ ├── ius_eval.py # IUS evaluation results (from eval_ius.py)
│ └── logs.py # TensorBoard logs (from train_epu.py )
├── scripts/ # Training and inference scripts
│ ├── eval_ius.py # Runs synthetic data evaluation with IUS
│ ├── infer_cb_vector.py # Estimates baseline feature contribution profiles
│ ├── infer_epu.py # Runs EPU-CNN inference/evaluation
│ └── train_epu.py # Trains EPU-CNN models
└── utils/ # Utility functions
├── callbacks.py # Training callbacks
├── config_utils.py # YAML/configuration utilities
├── early_stopping.py # Early stopping logic
├── eval_utils.py # Utilities for EPU-CNN evaluation scripts
├── metrics.py # Classification performance metrics
├── omega_parser.py # OmegaConf-based configuration parser
├── sanity_utils.py # Configuration validation and sanity checks
├── tensorboard.py # Tensorboard utilities
├── train_utils.py # Training setup and helping utilities
└── trainer.py # Main training loop implementation
Usage
1. Prepare Configuration
Create a YAML configuration file in configs/ with the following structure:
model:
num_subnetworks: 4 # set to 4, corresponds to number of perceptual feature maps,
num_classes: 1
epu_activation: "sigmoid"
subnetwork_config:
architecture: "base_one" # default ius backbone
input_channels: 1 # number of channels in perceptual feature decomposition, set to 1
base_channels: 32
fc_hidden_units: 64
pred_activation: "tanh"
data_params:
dataset_path: "../datasets/dataset_name"
images_extension: "jpg"
data_loading:
batch_size: 64
shuffle: true
num_workers: 0
pin_memory: false
persistent_workers: false
data_preprocessing:
data_mode: "rgb" # "rgb" or "grayscale"
data_parser: "filename" # "filename" or "folder" or "medmnist"
resize_dims: [128, 128]
medmnist_csv_file: None
label_mapping:
abnormal: 1
normal: 0
train_params:
mode: "binary"
loss: "binary_cross_entropy"
epochs: 200
optimizer: "sgd"
learning_rate: 0.001
momentum: 0.9
weight_decay: 0.001
early_stopping_patience: 30
early_stopping_monitor: "val_loss" # "val_loss" or "val_metrics.auc"
early_stopping_mode: "min" # "min" or "max"
log_dir: "../results/logs" # default parent path
checkpoint_dir: "../results/checkpoints" # default parent path
experiment_name: "ius_dataset_name" # desired experiment path
Key Points:
- For binary classification: Use
n_classes: 1,epu_activation: "sigmoid" - Adjust
input_sizeandbatch_sizebased on your GPU memory - Set
label_mappingaccording to your dataset classes
2. Supported Dataset Structures for EPU-CNN Training
EPU-CNN training supports multiple dataset organization patterns. Complete examples are provided below:
2.1 Filename-based Structure
datasets/
├── dataset_name/
├── train/
│ ├── abnormal_001.jpg
│ ├── abnormal_002.jpg
│ ├── normal_001.jpg
│ ├── normal_002.jpg
├── validation/
│ ├── abnormal_003.jpg
│ ├── normal_003.jpg
└── test/
├── abnormal_004.jpg
└── normal_004.jpg
Configuration Example:
data_params:
dataset_path: "../datasets/dataset_name"
images_extension: "jpg"
data_preprocessing:
data_mode: "rgb" # "rgb" or "grayscale"
data_parser: "filename"
resize_dims: [128, 128]
medmnist_csv_file: None
label_mapping:
abnormal: 1
normal: 0
Key Points:
- Requires a consistent organization across train/validation/test splits
- The validation folder name must be exactly "validation" (not "val" or other variants)
- Supported image formats include jpg, jpeg, png, etc, supported by the parser
- Class names defined in label_mapping must appear in filenames
2.2 Folder-based Structure
datasets/
├── dataset_name/
├── train/
│ ├── abnormal
│ │ ├── image_001.jpg
│ │ └── image_002.jpg
│ └── normal
│ ├── image_001.jpg
│ └── image_002.jpg
├── validation/
│ ├── abnormal
│ │ └── image_003.jpg
│ └── normal
│ └── image_003.jpg
└── test/
├── abnormal
│ └── image_004.jpg
└── normal
└── image_004.jpg
Configuration Example:
data_params:
dataset_path: "../datasets/dataset_name"
images_extension: "jpg"
data_preprocessing:
data_mode: "rgb" # "rgb" or "grayscale"
data_parser: "folder"
resize_dims: [128, 128]
medmnist_csv_file: None
label_mapping:
abnormal: 1
normal: 0
Key Points:
- Requires a consistent organization across train/validation/test splits
- The validation folder name must be exactly "validation" (not "val" or other variants)
- Supported image formats include jpg, jpeg, png, etc. supported by the parser
- Class names defined in label_mapping must match the class folder names
2.3 MedMNIST Benchmark Structure
datasets/
├── pneumoniamnist/
│ ├── pneumoniamnist.csv
│ ├── test_0_0.png
│ ├── test_1_1.png
│ ├── train_0_1.png
│ ├── train_1_0.png
│ ├── train_2_0.png
│ ├── train_3_1.png
│ ├── val_0_1.png
│ └── val_1_0.png
Configuration Example:
data_params:
dataset_path: "../datasets/pneumoniamnist"
images_extension: "png"
data_preprocessing:
data_mode: "rgb" # "rgb" or "grayscale"
data_parser: "folder"
resize_dims: [128, 128]
medmnist_csv_file: "../datasets/pneumoniamnist/pneumoniamnist.csv"
label_mapping:
pneumonia: 1
normal: 0
Key Points:
- Supports any 2D dataset included in the MedMNIST Benchmark collection.
- Requires the data in the original format provided by the MedMNIST benchmark
- The CSV file should be used as downloaded, without modification.
- Class names defined in label_mapping must match the official class names for the corresponding dataset provided in info
3. EPU-CNN Training
To train EPU-CNN, run one of the following commands after setting up a config.yaml file:
# Basic training
python scripts/train_epu.py --config_filepath configs/train_config.yaml
# Training with TensorBoard monitoring
python scripts/train_epu.py --config_filepath configs/train_config.yaml --tensorboard
The script saves trained model checkpoints and YAML training config in results/checkpoints under automatically generated name as: {experiment_name}{subnetwork_backbone}{run_id}_{timestamp}
When using the --tensorboard flag, the script automatically:
- launches TensorBoard as a subprocess
- sets up monitoring for the
logsdirectory - makes TensorBoard available at
http://localhost:6006 - enables real-time monitoring of training metrics and model graphs.
4. EPU-CNN Evaluation
To assess the classification performance of a trained EPU-CNN model run:
python scripts/infer_epu.py --experiment_folder_name ius_dataset_name_base_one_0000_timestamp
- Argument --experiment_folder_name contains the saved checkpoint and saved YAML config used during the EPU-CNN training.
- The classification performance report is saved in JSON format under results/classification_performance/{experiment_folder_name}
5. Feature contribution profile estimation
For each trained EPU-CNN model instance, the baseline feature contribution profile must be estimated only once using ether:
# Estimates all baseline feature contribution profiles
python scripts/infer_cb_vector.py --experiment_folder_name ius_dataset_name_base_one_0000_timestamp
# Estimates the baseline feature contribution profile of a single class
python scripts/infer_cb_vector.py --experiment_folder_name ius_dataset_name_base_one_0000_timestamp --cb_data normal
- The --experiment_folder_name contains the saved checkpoint and saved YAML config used during the EPU-CNN training.
- If argument --cb_data is not provided, the script estimates the baseline feature contribution profile for each class defined in saved config YAML (automatically retrieved from results/checkpoints based on --experiment_folder_name)
6. Synthetic image utility evaluation with IUS
To evaluate the utility of synthetic images using IUS, run:
# For synthetic dataset IUS evaluation
python scripts/eval_ius.py --experiment_folder_name ius_dataset_name_base_one_0000_timestamp --cb_vector_tag normal --synthetic_images datasets_synthetic/dataset_name/normal --synthetic_img_extension png
# For a single image IUS evaluation
python scripts/eval_ius.py --experiment_folder_name ius_dataset_name_base_one_0000_timestamp --cb_vector_tag normal --synthetic_images datasets_synthetic/dataset_name/normal/seed_000.png --synthetic_img_extension png
Both commands produce two outputs saved under results/ius_eval/{experiment_folder_name}
- A JSON report containing information about IUS evaluation.
- A CSV file containing the estimated IUS score for each synthetic image.
Key Points:
- Synthetic data should be stored under datasets_synthetic
- No specific internal folder structure is required; only a valid path to a folder of images is needed.
- A suggested organization example is provided below
- Supported image formats include jpg, jpeg, png, etc, supported by the parser
- Class labels are neither required nor used during IUS evaluation
Suggested synthetic data structure
datasets_synthetic/
├── dataset_name/
├── normal/
├── seed_000.png
└── seed_001.png
📄 Paper Abstract
Synthetic medical image data can unlock the potential of deep learning (DL)-based clinical decision support (CDS) systems through the creation of large scale, privacy-preserving, training sets. Despite the significant progress in this field, there is still a largely unanswered research question: “How can we quantitatively assess the similarity of a synthetically generated set of images with a set of real images in a given application domain?”. Today, answers to this question are mainly provided via user evaluation studies, inception-based measures, and the classification performance achieved on synthetic images. This paper proposes a novel measure to assess the similarity between synthetically generated and real sets of images, in terms of their utility for the development of DL based CDS systems. Inspired by generalized neural additive models, and unlike inception-based measures, the proposed measure is interpretable (Interpretable Utility Similarity, IUS), explaining why a synthetic dataset could be more useful than another one in the context of a CDS system based on clinically relevant image features. The experimental results on publicly available benchmark datasets from various color medical imaging modalities including endoscopic, dermoscopic and fundus imaging, indicate that selecting synthetic images of high utility similarity using IUS can result in relative improvements of up to 54.6% in terms of classification performance. The generality of IUS for synthetic data assessment is demonstrated also for grayscale X-ray and ultrasound imaging modalities. IUS implementation is available at https://github.com/innoisys/ius.
Citation
If you use this code or find our work useful in your research, please cite:
APA
P. Gatoula, G. Dimas and D. K. Iakovidis, "Interpretable Similarity of Synthetic Image Utility," in IEEE Transactions on Medical Imaging, doi: 10.1109/TMI.2026.3679527.
BibTeX
@article{
author = {Panagiota Gatoula and George Dimas and Dimitris K. Iakovidis},
title = {Interpretable Similarity of Synthetic Image Utility},
journal = {IEEE Transactions on Medical Imaging},
year = {2026},
publisher = {IEEE},
doi = {10.1109/TMI.2026.3679527}
}
Contact
Prof. Dimitris Iakovidis
Director of Biomedical Imaging Lab
University of Thessaly
License
This project is licensed under the MIT License.