QingCheng24/seamless_morph

Morph A/B Speaker Classifier (VideoMAE-Base)

A VideoMAE-Base video classifier that predicts a speaker's behavioral "Morph" label (Morph A vs. Morph B) from a short clip of them talking in a two-person conversation. Fine-tuned on Meta's Seamless Interaction dataset.

Code / training pipeline: see the accompanying GitHub repository.

Files

File	Description
best_acc.ckpt	PyTorch-Lightning checkpoint (~1 GB), best val/acc
config.yaml	exact training config for this run
MODEL_CARD.md	this card

Model

• Base model: MCG-NJU/videomae-base-finetuned-kinetics (~86 M params), loaded via transformers.VideoMAEForVideoClassification. The K400 400-way head is replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned.
• Input: 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224, ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position embeddings scale from the pretrained 16 frames to 64 with no interpolation.

Training

• Data: ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B.
• Split: random per-clip 80/20, stratified by class.
• Optimizer: AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05.
• Schedule: 10-epoch warmup + cosine decay to 1e-6; early stopping on val/acc.
• Loss: class-weighted cross-entropy.

Results

Metric	Value
Val accuracy	~99.6%
Val F1	~99.5%

Intended use

Research baseline for whether the behavioral "Morph" distinction is decodable from raw video.

Usage

huggingface-cli download QingCheng24/seamless_morph --local-dir weights

python train_videomae.py --test --ckpt_path weights/best_acc.ckpt

License

Released under CC BY-NC 4.0, inheriting the non-commercial terms of the Seamless Interaction dataset and the Morph A/B annotations it was trained on.