File size: 2,204 Bytes
89d5b2b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
license: cc-by-nc-4.0
base_model: MCG-NJU/videomae-base-finetuned-kinetics
tags:
- video-classification
- videomae
- seamless-interaction
pipeline_tag: video-classification
---
# Morph A/B Speaker Classifier (VideoMAE-Base)
A VideoMAE-Base video classifier that predicts a speaker's behavioral **"Morph"
label (Morph A vs. Morph B)** from a short clip of them talking in a two-person
conversation. Fine-tuned on
[Meta's Seamless Interaction dataset](https://github.com/facebookresearch/seamless_interaction).
**Code / training pipeline:** see the accompanying GitHub repository.
## Files
| File | Description |
|------|-------------|
| `best_acc.ckpt` | PyTorch-Lightning checkpoint (~1 GB), best `val/acc` |
| `config.yaml` | exact training config for this run |
| `MODEL_CARD.md` | this card |
## Model
- **Base model:** `MCG-NJU/videomae-base-finetuned-kinetics` (~86 M params), loaded
via `transformers.VideoMAEForVideoClassification`. The K400 400-way head is
replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned.
- **Input:** 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224,
ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position
embeddings scale from the pretrained 16 frames to 64 with no interpolation.
## Training
- **Data:** ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B.
- **Split:** random **per-clip** 80/20, stratified by class.
- **Optimizer:** AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05.
- **Schedule:** 10-epoch warmup + cosine decay to 1e-6; early stopping on `val/acc`.
- **Loss:** class-weighted cross-entropy.
## Results
| Metric | Value |
|--------|------:|
| Val accuracy | ~99.6% |
| Val F1 | ~99.5% |
## Intended use
Research baseline for whether the behavioral "Morph" distinction is decodable
from raw video.
## Usage
```bash
huggingface-cli download QingCheng24/seamless_morph --local-dir weights
python train_videomae.py --test --ckpt_path weights/best_acc.ckpt
```
## License
Released under **CC BY-NC 4.0**, inheriting the non-commercial terms of the
Seamless Interaction dataset and the Morph A/B annotations it was trained on.
|