README.md

---
license: mit
tags:
  - active-speaker-detection
  - audio-visual
  - speaker-diarization
  - streaming
  - pytorch
  - pytorch-lightning
---

# CAST-ASD

Pretrained weights for **CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection**.

Pratham Gupta, Haotian Qi, Tushar Sandhan, Gabriel Skantze

[Paper (arXiv:2610.14676)](https://arxiv.org/abs/2610.14676) · [Code](https://github.com/prathamg007/CAST-ASD) · [Project page and interactive demo](https://prathamg007.github.io/CAST-ASD/)

CAST-ASD decides, for every visible face in every video frame, whether that person is speaking, without reading any future frame. It adds one CAST block to a causal per-face backbone: a speaker pass attends across the faces present at the current frame, and a time pass attends causally over each face's own past. The block has 1.57M parameters, independent of the number of faces. On MSDWild it removes 32% of the competing-speaker false alarms of the causal per-face backbone it extends.

## Files

All four checkpoints are CAST ×1 (one block). Each needs its config, included here and in the code repository.

| File | Config | Seed | Validation mAP | Paper (mean of 3 seeds) |
|---|---|:---:|---|---|
| `weights/msdwild_cast.ckpt` | `configs/msdwild/cast.yaml` | 42 | few.val 97.01, many.val 90.66 | few.val 96.99, many.val 90.56 |
| `weights/msdwild_cast_offline.ckpt` | `configs/msdwild/cast_offline.yaml` | 1 | few.val 97.55, many.val 91.77 | few.val 97.51, many.val 91.70 |
| `weights/ava_cast.ckpt` | `configs/ava/cast.yaml` | 2 | official 91.21 | official 91.07 |
| `weights/ava_cast_offline.ckpt` | `configs/ava/cast_offline.yaml` | 1 | official 94.31 | official 94.20 |

- `msdwild_*` are trained on MSDWild (few-speaker training split); `ava_*` on AVA-ActiveSpeaker.
- `*_offline` checkpoints use the same architecture without the causal constraints and read the whole clip.
- Each model was trained with seeds 42, 1 and 2; the released file is the best of the three on the selection metric (few.val mAP on MSDWild, official mAP on AVA). The paper reports the mean over the three seeds.
- MSDWild checkpoints are from the final (12th) epoch; AVA checkpoints are the best epoch under early stopping.
- The frozen Mimi audio encoder is not stored in these files. It is downloaded from `kyutai/moshiko-pytorch-bf16` and merged in when a checkpoint loads. Optimizer state is stripped, so the files evaluate but cannot resume training.

## Usage

The checkpoints are PyTorch Lightning files for the code in the [GitHub repository](https://github.com/prathamg007/CAST-ASD).

```bash
git clone https://github.com/prathamg007/CAST-ASD.git
cd CAST-ASD
pip install -e ".[mimi]"
hf download prathamg007/CAST-ASD --include "weights/*" --local-dir .
python train.py --config configs/msdwild/cast.yaml --test --checkpoint weights/msdwild_cast.ckpt
```

Or fetch a single file from Python:

```python
from huggingface_hub import hf_hub_download

path = hf_hub_download("prathamg007/CAST-ASD", "weights/msdwild_cast.ckpt")
```

Data preparation for MSDWild and AVA-ActiveSpeaker is described in the code repository.

## License

MIT, the same as the code. The models were trained on AVA-ActiveSpeaker and MSDWild, so use them in line with those datasets' terms (MSDWild is distributed under its own license agreement).

## Citation

```bibtex
@misc{gupta2026castasd,
  title         = {{CAST-ASD}: Causal Attention over Speakers and Time for Online Active Speaker Detection},
  author        = {Gupta, Pratham and Qi, Haotian and Sandhan, Tushar and Skantze, Gabriel},
  year          = {2026},
  eprint        = {2610.14676},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2610.14676},
  url           = {https://arxiv.org/abs/2610.14676}
}
```