metadata
license: cc-by-nc-4.0
base_model:
- microsoft/beit-base-patch16-384
pipeline_tag: zero-shot-image-classification
tags:
- medical
ECG‑CLIP‑BEiT‑Base‑384
A CLIP‑based model fine‑tuned on paired ECG images and echo reports to generate 512‑dim embeddings and perform zero‑shot classification of echocardiographic phenotypes. Read more here: https://github.com/CarDS-Yale/target-ai-shared
Description
This model uses:
- Vision encoder:
microsoft/beit-base-patch16-384 - Text encoder: custom ByteLevel BPE tokenizer (vocab_size=16 030)
- Projection dimension: 512
- Trained on: paired ECG images and corresponding echo report text
- Tasks:
- Produce image embeddings for ECG images
- Zero‑shot classification of phenotypes (e.g., LV dysfunction, AS) via cosine similarity to precomputed centroids
Setup & Usage
Clone the repository and go to demo directory:
git clone https://github.com/CarDS-Yale/target-ai-shared.git
cd target-ai-shared
Make inference environment::
conda env create -f ./environment_files/ecg_image_vit_light.yml
Run inference for a set of images:: You can run quick inference on a folder of ECG images by accessing the model's weights on HuggingFace. In the command below, the following can be adjusted:
- hf_repo: this points to the HF repo of the published model and weights
- image_dir: point to a folder containing ECG images in the 4 standard layouts - as shown below
- centroid_csv: point to the csv containing reference embeddings/centroids for cases/controls - we provide examples from our training set and EchoNext (Elias, P., & Finer, J. (2025). EchoNext: A Dataset for Detecting Echocardiogram-Confirmed Structural Heart Disease from ECGs (version 1.1.0). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/3ykd-bf14) ----- e.g. ./demo/reference_embeddings/echonext_reference_centroids.csv or ./demo/reference_embeddings/ynhhs_reference_centroids.csv
- output_csv: path where the output will be stored
conda activate ecg_image_vit_light
python ./demo/zero_shot_from_reference_embedding.py \
--hf_repo "CarDSLab/ecg-clip-beit-base-384" \
--image_dir "./demo/online_ecg_images_credit_to_liftl" \
--centroid_csv "./demo/reference_embeddings/ynhhs_reference_centroids.csv" \
--output_csv "./demo/output/output.csv" \
--batch_size 32 \
--device cuda
Output (output_csv): one row per image×label with columns:
image_id,label,cosine_similarity_to_case,delta_cosine_similarity_case_minus_control,embedding
Citation
@article{oikonomou2025targetecg,
title = {TARGET-AI: a foundational approach for the targeted deployment of artificial intelligence electrocardiography in the electronic health record},
author = {Oikonomou, Evangelos K. and Khera, Rohan and others},
journal = {medRxiv},
year = {2025},
doi = {10.1101/2025.08.25.25334266},
note = {Preprint}
}