---
license: cc-by-nc-4.0
tags:
- echocardiography
- video
- masked-autoencoder
- self-supervised
- cardiac
pipeline_tag: video-classification
---

# EchoFM: A Video Vision Foundation Model for Echocardiography

ViT-L video masked autoencoder pretrained on ~41k apical echocardiogram clips with a
cardiac-cycle-aware objective:

`L = L_recon (norm-pix, 75% spatio-temporally consistent masking) + L_triplet + L_cycle-KL`

- **triplet**: positives/negatives chosen by a pixel-space cycle-similarity prior (hard mining, cosine margin 0.2)
- **cycle-KL**: per-anchor embedding-similarity distributions distilled toward the pixel prior with the
  static (anatomy) component removed — this makes the embeddings cardiac-phase-aware

Code, training pipeline, and diagnostics: https://github.com/SekeunKim/EchoFM

## Checkpoint

`echofm_vitl.pth` — contains `{"model": state_dict, "model_args": dict}` (1.4 GB).

Validation on held-out clips:

| metric | value |
|---|---|
| embedding-vs-pixel cycle correlation r | 0.977 |
| phase contrast (same-phase minus opposite-phase similarity) | 0.69 (positive on 100% of clips) |
| masked PSNR (75% masking) | 24.0 dB |

## Usage

```python
import torch
from huggingface_hub import hf_hub_download
from EchoFM import models_mae  # from the GitHub repo

weights = hf_hub_download(repo_id="sekeun/EchoFM", filename="echofm_vitl.pth")
ckpt = torch.load(weights, map_location="cpu")
model = models_mae.mae_vit_large_patch16(**{
    k: ckpt["model_args"][k] for k in
    ["num_frames", "t_patch_size", "pred_t_dim", "sep_pos_embed", "cls_embed", "norm_pix_loss"]
})
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

# imgs: [B, 3, 32, 224, 224] in [0, 1]
latent, _, _ = model.forward_encoder(imgs, mask_ratio=0.0)      # [B, 8*196, 1024] tokens
cls_stack = torch.stack(model.forward_prj(latent), dim=1)       # [B, 8, 1024] per-frame (phase) embeddings
video_emb = latent.mean(dim=1)                                  # [B, 1024] video embedding
```

See `notebooks/echofm_usage.ipynb` in the GitHub repo for feature extraction, masked
reconstruction, and periodicity verification examples.