---
license: cc-by-nc-4.0
base_model: MCG-NJU/videomae-base-finetuned-kinetics
tags:
  - video-classification
  - videomae
  - seamless-interaction
pipeline_tag: video-classification
---

# Morph A/B Speaker Classifier (VideoMAE-Base)

A VideoMAE-Base video classifier that predicts a speaker's behavioral **"Morph"
label (Morph A vs. Morph B)** from a short clip of them talking in a two-person
conversation. Fine-tuned on
[Meta's Seamless Interaction dataset](https://github.com/facebookresearch/seamless_interaction).

**Code / training pipeline:** see the accompanying GitHub repository.

## Files

| File | Description |
|------|-------------|
| `best_acc.ckpt` | PyTorch-Lightning checkpoint (~1 GB), best `val/acc` |
| `config.yaml` | exact training config for this run |
| `MODEL_CARD.md` | this card |

## Model

- **Base model:** `MCG-NJU/videomae-base-finetuned-kinetics` (~86 M params), loaded
  via `transformers.VideoMAEForVideoClassification`. The K400 400-way head is
  replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned.
- **Input:** 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224,
  ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position
  embeddings scale from the pretrained 16 frames to 64 with no interpolation.

## Training

- **Data:** ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B.
- **Split:** random **per-clip** 80/20, stratified by class.
- **Optimizer:** AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05.
- **Schedule:** 10-epoch warmup + cosine decay to 1e-6; early stopping on `val/acc`.
- **Loss:** class-weighted cross-entropy.

## Results

| Metric | Value |
|--------|------:|
| Val accuracy | ~99.6% |
| Val F1 | ~99.5% |

## Intended use

Research baseline for whether the behavioral "Morph" distinction is decodable
from raw video.

## Usage

```bash
huggingface-cli download QingCheng24/seamless_morph --local-dir weights

python train_videomae.py --test --ckpt_path weights/best_acc.ckpt
```

## License

Released under **CC BY-NC 4.0**, inheriting the non-commercial terms of the
Seamless Interaction dataset and the Morph A/B annotations it was trained on.