| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - echocardiography |
| - video |
| - masked-autoencoder |
| - self-supervised |
| - cardiac |
| pipeline_tag: video-classification |
| --- |
| |
| # EchoFM: A Video Vision Foundation Model for Echocardiography |
|
|
| ViT-L video masked autoencoder pretrained on ~41k apical echocardiogram clips with a |
| cardiac-cycle-aware objective: |
|
|
| `L = L_recon (norm-pix, 75% spatio-temporally consistent masking) + L_triplet + L_cycle-KL` |
|
|
| - **triplet**: positives/negatives chosen by a pixel-space cycle-similarity prior (hard mining, cosine margin 0.2) |
| - **cycle-KL**: per-anchor embedding-similarity distributions distilled toward the pixel prior with the |
| static (anatomy) component removed — this makes the embeddings cardiac-phase-aware |
|
|
| Code, training pipeline, and diagnostics: https://github.com/SekeunKim/EchoFM |
|
|
| ## Checkpoint |
|
|
| `echofm_vitl.pth` — contains `{"model": state_dict, "model_args": dict}` (1.4 GB). |
|
|
| Validation on held-out clips: |
|
|
| | metric | value | |
| |---|---| |
| | embedding-vs-pixel cycle correlation r | 0.977 | |
| | phase contrast (same-phase minus opposite-phase similarity) | 0.69 (positive on 100% of clips) | |
| | masked PSNR (75% masking) | 24.0 dB | |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from EchoFM import models_mae # from the GitHub repo |
| |
| weights = hf_hub_download(repo_id="sekeun/EchoFM", filename="echofm_vitl.pth") |
| ckpt = torch.load(weights, map_location="cpu") |
| model = models_mae.mae_vit_large_patch16(**{ |
| k: ckpt["model_args"][k] for k in |
| ["num_frames", "t_patch_size", "pred_t_dim", "sep_pos_embed", "cls_embed", "norm_pix_loss"] |
| }) |
| model.load_state_dict(ckpt["model"], strict=False) |
| model.eval() |
| |
| # imgs: [B, 3, 32, 224, 224] in [0, 1] |
| latent, _, _ = model.forward_encoder(imgs, mask_ratio=0.0) # [B, 8*196, 1024] tokens |
| cls_stack = torch.stack(model.forward_prj(latent), dim=1) # [B, 8, 1024] per-frame (phase) embeddings |
| video_emb = latent.mean(dim=1) # [B, 1024] video embedding |
| ``` |
|
|
| ## ED/ES and cardiac-cycle extraction |
|
|
| `echofm_phase.py` (in this repo) provides ready-to-use phase utilities — cycle length, |
| heart rate, ED/ES frame detection (no model needed), and embedding-based same-phase |
| retrieval: |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| import importlib.util |
| |
| spec = importlib.util.spec_from_file_location( |
| "echofm_phase", hf_hub_download("sekeun/EchoFM", "echofm_phase.py")) |
| phase = importlib.util.module_from_spec(spec); spec.loader.exec_module(phase) |
| |
| # clip: float tensor [3, T, H, W] in [0, 1] |
| info = phase.detect_ed_es(clip, fps=30) |
| # {'ed_frames': [6, 30], 'es_frame': 13, 'cycle_frames': 24, 'hr_bpm': 75.0, ...} |
| |
| z = phase.phase_embeddings(model, clip) # [8, 1024] per-timestep phase embeddings |
| match = phase.same_phase_frame(model, clip, frame=info["ed_frames"][0]) |
| # {'match_frame': 30, ...} — retrieves the same phase in the next cycle |
| ``` |
|
|
| See `notebooks/echofm_usage.ipynb` in the GitHub repo for more examples. |
|
|
