metadata
license: cc-by-nc-4.0
tags:
- echocardiography
- video
- masked-autoencoder
- self-supervised
- cardiac
pipeline_tag: video-classification
EchoFM: A Video Vision Foundation Model for Echocardiography
ViT-L video masked autoencoder pretrained on ~41k apical echocardiogram clips with a cardiac-cycle-aware objective:
L = L_recon (norm-pix, 75% spatio-temporally consistent masking) + L_triplet + L_cycle-KL
- triplet: positives/negatives chosen by a pixel-space cycle-similarity prior (hard mining, cosine margin 0.2)
- cycle-KL: per-anchor embedding-similarity distributions distilled toward the pixel prior with the static (anatomy) component removed — this makes the embeddings cardiac-phase-aware
Code, training pipeline, and diagnostics: https://github.com/SekeunKim/EchoFM
Checkpoint
echofm_vitl.pth — contains {"model": state_dict, "model_args": dict} (1.4 GB).
Validation on held-out clips:
| metric | value |
|---|---|
| embedding-vs-pixel cycle correlation r | 0.977 |
| phase contrast (same-phase minus opposite-phase similarity) | 0.69 (positive on 100% of clips) |
| masked PSNR (75% masking) | 24.0 dB |
Usage
import torch
from huggingface_hub import hf_hub_download
from EchoFM import models_mae # from the GitHub repo
weights = hf_hub_download(repo_id="sekeun/EchoFM", filename="echofm_vitl.pth")
ckpt = torch.load(weights, map_location="cpu")
model = models_mae.mae_vit_large_patch16(**{
k: ckpt["model_args"][k] for k in
["num_frames", "t_patch_size", "pred_t_dim", "sep_pos_embed", "cls_embed", "norm_pix_loss"]
})
model.load_state_dict(ckpt["model"], strict=False)
model.eval()
# imgs: [B, 3, 32, 224, 224] in [0, 1]
latent, _, _ = model.forward_encoder(imgs, mask_ratio=0.0) # [B, 8*196, 1024] tokens
cls_stack = torch.stack(model.forward_prj(latent), dim=1) # [B, 8, 1024] per-frame (phase) embeddings
video_emb = latent.mean(dim=1) # [B, 1024] video embedding
See notebooks/echofm_usage.ipynb in the GitHub repo for feature extraction, masked
reconstruction, and periodicity verification examples.