| --- |
| license: cc-by-nc-4.0 |
| base_model: MCG-NJU/videomae-base-finetuned-kinetics |
| tags: |
| - video-classification |
| - videomae |
| - seamless-interaction |
| pipeline_tag: video-classification |
| --- |
| |
| # Morph A/B Speaker Classifier (VideoMAE-Base) |
|
|
| A VideoMAE-Base video classifier that predicts a speaker's behavioral **"Morph" |
| label (Morph A vs. Morph B)** from a short clip of them talking in a two-person |
| conversation. Fine-tuned on |
| [Meta's Seamless Interaction dataset](https://github.com/facebookresearch/seamless_interaction). |
|
|
| **Code / training pipeline:** see the accompanying GitHub repository. |
|
|
| ## Files |
|
|
| | File | Description | |
| |------|-------------| |
| | `best_acc.ckpt` | PyTorch-Lightning checkpoint (~1 GB), best `val/acc` | |
| | `config.yaml` | exact training config for this run | |
| | `MODEL_CARD.md` | this card | |
|
|
| ## Model |
|
|
| - **Base model:** `MCG-NJU/videomae-base-finetuned-kinetics` (~86 M params), loaded |
| via `transformers.VideoMAEForVideoClassification`. The K400 400-way head is |
| replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned. |
| - **Input:** 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224, |
| ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position |
| embeddings scale from the pretrained 16 frames to 64 with no interpolation. |
|
|
| ## Training |
|
|
| - **Data:** ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B. |
| - **Split:** random **per-clip** 80/20, stratified by class. |
| - **Optimizer:** AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05. |
| - **Schedule:** 10-epoch warmup + cosine decay to 1e-6; early stopping on `val/acc`. |
| - **Loss:** class-weighted cross-entropy. |
|
|
| ## Results |
|
|
| | Metric | Value | |
| |--------|------:| |
| | Val accuracy | ~99.6% | |
| | Val F1 | ~99.5% | |
|
|
| ## Intended use |
|
|
| Research baseline for whether the behavioral "Morph" distinction is decodable |
| from raw video. |
|
|
| ## Usage |
|
|
| ```bash |
| huggingface-cli download QingCheng24/seamless_morph --local-dir weights |
| |
| python train_videomae.py --test --ckpt_path weights/best_acc.ckpt |
| ``` |
|
|
| ## License |
|
|
| Released under **CC BY-NC 4.0**, inheriting the non-commercial terms of the |
| Seamless Interaction dataset and the Morph A/B annotations it was trained on. |
|
|
