EchoFM: A Video Vision Foundation Model for Echocardiography

ViT-L video masked autoencoder pretrained on ~41k apical echocardiogram clips with a cardiac-cycle-aware objective:

L = L_recon (norm-pix, 75% spatio-temporally consistent masking) + L_triplet + L_cycle-KL

• triplet: positives/negatives chosen by a pixel-space cycle-similarity prior (hard mining, cosine margin 0.2)
• cycle-KL: per-anchor embedding-similarity distributions distilled toward the pixel prior with the static (anatomy) component removed — this makes the embeddings cardiac-phase-aware

Code, training pipeline, and diagnostics: https://github.com/SekeunKim/EchoFM

Checkpoint

echofm_vitl.pth — contains {"model": state_dict, "model_args": dict} (1.4 GB).

Validation on held-out clips:

metric	value
embedding-vs-pixel cycle correlation r	0.977
phase contrast (same-phase minus opposite-phase similarity)	0.69 (positive on 100% of clips)
masked PSNR (75% masking)	24.0 dB

Usage

import torch
from huggingface_hub import hf_hub_download
from EchoFM import models_mae  # from the GitHub repo

weights = hf_hub_download(repo_id="sekeun/EchoFM", filename="echofm_vitl.pth")
ckpt = torch.load(weights, map_location="cpu")
model = models_mae.mae_vit_large_patch16(**{
    k: ckpt["model_args"][k] for k in
    ["num_frames", "t_patch_size", "pred_t_dim", "sep_pos_embed", "cls_embed", "norm_pix_loss"]
})
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

# imgs: [B, 3, 32, 224, 224] in [0, 1]
latent, _, _ = model.forward_encoder(imgs, mask_ratio=0.0)      # [B, 8*196, 1024] tokens
cls_stack = torch.stack(model.forward_prj(latent), dim=1)       # [B, 8, 1024] per-frame (phase) embeddings
video_emb = latent.mean(dim=1)                                  # [B, 1024] video embedding

See notebooks/echofm_usage.ipynb in the GitHub repo for feature extraction, masked reconstruction, and periodicity verification examples.