olaflaitinen/ecg-clip-scalogram-spectrogram

ECG-CLIP: Scalogram versus Spectrogram Zero-Shot Cardiac Classification

ECG-CLIP is a contrastive vision-language model that aligns 12-lead electrocardiogram time-frequency images with clinical text, enabling zero-shot classification of cardiac conditions without task-specific labeled training. This repository releases two models that are identical except for the input representation, one built from scalograms (continuous wavelet transform) and one from spectrograms (short-time Fourier transform). The pair supports a controlled comparison of the two representations under a fixed encoder, prompt set, and training budget. These are the companion artifacts for the study submitted to Biomedical Signal Processing and Control.

Model details

• Developed by: Mehmet Fidan, Derya Umut Kulali, Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov
• Affiliations: Eskisehir Technical University; Metropolia University of Applied Sciences, Helsinki
• Model type: Contrastive vision-language (CLIP-style) model for 12-lead ECG
• Input modalities: ECG time-frequency image and clinical text prompt
• Prompt language: English
• License: CC-BY-4.0
• Repository: https://huggingface.co/olaflaitinen/ecg-clip-scalogram-spectrogram
• DOI: https://doi.org/10.57967/hf/9559
• Paper: Comparative Evaluation of ECG Scalograms and Spectrograms for CLIP-Based Zero-Shot Classification of Cardiac Conditions (under review)
• Point of contact: yunus.imanov@metropolia.fi

Model variants

File	Image representation	Vision backbone	Text encoder	Parameters
main_scal.safetensors	Scalogram (CWT, cmor1.5-1.0)	ResNet-50	Bio_ClinicalBERT	approximately 133 M
main_spec.safetensors	Spectrogram (STFT, Hann window)	ResNet-50	Bio_ClinicalBERT	approximately 133 M

Each model contains an image encoder, a text encoder, and a shared 512-dimensional projection space. The two towers are trained jointly with a symmetric InfoNCE contrastive objective and a learnable temperature.

Intended use

Direct use

• Zero-shot classification of PTB-XL diagnostic superclasses (NORM, MI, STTC, CD, HYP) and subclasses from a 12-lead ECG, by comparing the image embedding against text prompt prototypes.
• Research on label-efficient ECG interpretation and on the effect of time-frequency representation choice for physiological signals.

Downstream use

• Feature extraction followed by a lightweight linear probe when a small labeled set is available.
• A starting point for domain adaptation to other 12-lead ECG cohorts.

Out of scope

• Clinical decision-making of any kind. These models are research artifacts and must not be used for diagnosis, triage, screening, or treatment.
• Deployment on single-lead or reduced-lead wearable ECG without retraining, since the models expect a 12-lead montage.

How to use

import numpy as np
from modeling_ecgclip import from_pretrained, zero_shot

# signal: numpy array of shape [T, 12] sampled at 100 Hz,
# lead order I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, V6
model = from_pretrained("main_scal.safetensors", representation="scalogram")
probs = zero_shot(model, signal, fs=100)
print(probs)  # {"NORM": ..., "MI": ..., "STTC": ..., "CD": ..., "HYP": ...}

The model definition, preprocessing, and loader are provided in modeling_ecgclip.py in this repository.

Training details

Training data

• Dataset: PTB-XL version 1.0.3 (Wagner et al., 2020), distributed by PhysioNet under CC-BY 4.0.
• Records: 12-lead, 10 seconds, downsampled to 100 Hz for the released models.
• Splits: stratified folds 1 to 8 for training, fold 9 for validation, fold 10 for testing.
• No PTB-XL waveforms are redistributed in this repository. Download the data from PhysioNet.

Preprocessing

• Per-lead high-pass filtering at 0.5 Hz and a 50 Hz notch filter, followed by per-lead standardization.
• Scalogram: continuous wavelet transform with the complex Morlet wavelet cmor1.5-1.0, 32 geometrically spaced scales.
• Spectrogram: short-time Fourier transform with a Hann window, retaining the 0.5 to 40 Hz diagnostic band.
• The 12 per-lead maps are tiled into a 4 by 3 montage and resized to 224 by 224, then replicated to three channels and normalized with ImageNet statistics.

Training configuration

Setting	Value
Objective	Symmetric InfoNCE (CLIP)
Embedding dimension	512
Optimizer	AdamW
Peak learning rate	1e-4
Weight decay	0.2
Warmup ratio	0.1
Schedule	Cosine decay
Epochs	50
Batch size per GPU	128
Initial temperature	0.07
Text encoder	emilyalsentzer/Bio_ClinicalBERT
Vision backbone	ResNet-50 (ImageNet pretrained)
Random seed	2025

Evaluation

All metrics are computed on the PTB-XL test set (fold 10) and macro-averaged over classes. AUROC is the primary metric; AUPRC, F1, and expected calibration error (ECE) are also reported.

Zero-shot classification

Representation	Level	AUROC	AUPRC	F1	ECE
Spectrogram	Superclass	0.526	0.285	0.384	0.324
Scalogram	Superclass	0.590	0.321	0.421	0.247
Spectrogram	Subclass	0.611	0.119	0.151	0.390
Scalogram	Subclass	0.689	0.124	0.159	0.332

Linear probe

Representation	Level	AUROC	AUPRC	F1	ECE
Spectrogram	Superclass	0.774	0.538	0.541	0.150
Scalogram	Superclass	0.775	0.531	0.536	0.157
Spectrogram	Subclass	0.739	0.192	0.212	0.061
Scalogram	Subclass	0.749	0.198	0.228	0.058

Paired comparison (scalogram minus spectrogram)

Estimated with a paired bootstrap over 2000 resamples on the shared test set.

Level	Delta AUROC	95% CI	p value
Superclass	+0.063	[0.048, 0.078]	< 0.001
Subclass	+0.078	[0.048, 0.107]	< 0.001

Per-class zero-shot AUROC (superclass)

Class	Spectrogram	Scalogram
NORM	0.627	0.447
MI	0.487	0.613
STTC	0.572	0.705
CD	0.441	0.555
HYP	0.504	0.627

Representation and encoder ablations (zero-shot superclass AUROC)

Configuration	Spectrogram	Scalogram
Main (100 Hz, ResNet-50)	0.526	0.590
Abbreviation expansion removed	TBD	TBD
500 Hz input	0.516	0.581
ViT-B/16 backbone	0.498	0.563
Alternative STFT window (128)	TBD	not applicable
Raw-waveform baseline (1D ResNet)	TBD	not applicable

Subgroup analysis (zero-shot superclass AUROC)

Subgroup	Spectrogram	Scalogram
Male	TBD	TBD
Female	TBD	TBD
Age 50 and under	TBD	TBD
Age 51 to 65	TBD	TBD
Age 66 and over	TBD	TBD

External validation (Chapman-Shaoxing, zero-shot superclass AUROC)

Class	Spectrogram	Scalogram
NORM	TBD	TBD
MI	TBD	TBD
STTC	TBD	TBD
CD	TBD	TBD
HYP	TBD	TBD
Macro	TBD	TBD

Limitations and biases

• Absolute zero-shot AUROC is modest. This is a controlled representation comparison, not a deployable classifier.
• The scalogram NORM (normal) AUROC is below 0.5. The model should therefore be used to refer rather than to rule out, and it should be paired with a dedicated normal versus abnormal detector before any downstream use.
• Training uses a single-center European cohort (PTB-XL), so external and cross-population generalization is limited. External validation numbers in this card will quantify this once available.
• Prompt wording affects zero-shot performance. The released prompts are fixed for reproducibility and were not tuned per class on the test set.
• Subgroup performance may vary by sex and age. See the subgroup table once populated.

Environmental and compute information

• Training hardware: two NVIDIA T4 GPUs (distributed data parallel).
• Approximate training time per released model: TBD.
• Estimated total compute for the full study: TBD.

Citation

@article{fidan_ecgclip_2026,
  title   = {Comparative Evaluation of ECG Scalograms and Spectrograms for CLIP-Based Zero-Shot Classification of Cardiac Conditions},
  author  = {Fidan, Mehmet and Kulali, Derya Umut and Laitinen-Fredriksson Lundstrom-Imanov, Gustav Olaf Yunus},
  journal = {Biomedical Signal Processing and Control},
  year    = {2026},
  note    = {Under review}
}

A dataset citation is also required when using the models:

@article{wagner2020ptbxl,
  title   = {PTB-XL, a large publicly available electrocardiography dataset},
  author  = {Wagner, Patrick and Strodthoff, Nils and Bousseljot, Ralf-Dieter and Kreiseler, Dieter and Lunze, Fatima I. and Samek, Wojciech and Schaeffter, Tobias},
  journal = {Scientific Data},
  volume  = {7},
  number  = {1},
  pages   = {154},
  year    = {2020}
}

When citing the released model weights specifically, also reference the HuggingFace artifact DOI: https://doi.org/10.57967/hf/9559.

Acknowledgements

This work builds on PTB-XL (CC-BY 4.0), Bio_ClinicalBERT (MIT license), and ImageNet-pretrained backbones from the timm library.

Model card contact

For questions about these models, contact the corresponding author at yunus.imanov@metropolia.fi or open a discussion on the repository.