README.md

---
license: apache-2.0
tags:
- medical-imaging
- videomae
- risk-prediction
pipeline_tag: video-classification
---

# EchoFuture

**EchoFuture** is a video masked autoencoder (VideoMAE) pre-trained on 2.5 million echocardiogram videos. 
**EchoFuture-HFrEF** is fine-tuned from EchoFuture to predict 10-year risk of heart failure with reduced ejection fraction (HFrEF) from a single A2C or A4C echocardiogram clip.

<p align="center">
  <img src="EchoFuture.gif" width="100%" />
</p>

## Repository structure

```
VoyagerWSH/EchoFuture/
├── pretrained/          # EchoFuture foundation model
│   ├── config.json
│   ├── model.safetensors
│   └── preprocessor_config.json
└── hfref/               # EchoFuture-HFrEF risk model
    └── echofuture_hfref.pth
```

## Usage

### EchoFuture (pre-trained backbone)

Load the VideoMAE encoder pre-trained on echocardiograms:

```python
from transformers import VideoMAEForPreTraining

model = VideoMAEForPreTraining.from_pretrained(
    "VoyagerWSH/EchoFuture",
    subfolder="pretrained",
    attn_implementation="sdpa",
)
```

### EchoFuture-HFrEF (fine-tuned risk model)

Load the 10-year HFrEF risk prediction model. Requires the `echofuture` package from the [code repository](https://github.com/<your-org>/EchoFuture).

```python
import torch
from huggingface_hub import hf_hub_download
from echofuture.model import EchoFutureHFrEF, strip_module_prefix

model = EchoFutureHFrEF(num_followups=10)
weights = hf_hub_download(
    "VoyagerWSH/EchoFuture", filename="echofuture_hfref.pth", subfolder="hfref"
)
sd = torch.load(weights, map_location="cpu", weights_only=False)
if isinstance(sd, dict) and "state_dict" in sd:
    sd = sd["state_dict"]
sd = strip_module_prefix(sd)
model.load_state_dict(sd, strict=True)
model.eval()
```

**Input:** `(B, C=3, T=16, H=224, W=224)` — 16 frames at 15 fps, 224 x 224, RGB.

**Output:** `(B, 10)` logits — one per yearly follow-up (years 1–10). Apply `sigmoid` to obtain cumulative risk probabilities.

### Cohort-scale inference

The code repository provides a CLI for batch inference on a cohort CSV:

```bash
pip install -e .
echofuture-infer --data /path/to/cohort.csv --output-dir ./output
```

See the [code repository](https://github.com/VoyagerWSH/EchoFuture) for dataset format, configuration, and evaluation details.

## Model details

| | |
|---|---|
| **Architecture** | VideoMAE-base (ViT-B, 86M params) |
| **Pre-training** | Masked autoencoding on echocardiogram videos |
| **Fine-tuning** | Cumulative HFrEF risk prediction (BCE loss with censoring masks) |
| **Input** | 16 frames at 15 fps, 224 x 224, A2C/A4C views |
| **Output** | 10-year cumulative HFrEF risk (yearly intervals) |

## Citation

Under review.