You can deploy this model and start earning money today!

QingCheng24/
seamless_morph

video-classification

videomae

seamless-interaction

MCG-NJU/videomae-base-finetuned-kinetics

finetune

cc-by-nc-4.0

Views

No views yet

Morph A/B Speaker Classifier (VideoMAE-Base)

A VideoMAE-Base video classifier that predicts a speaker's behavioral "Morph" label (Morph A vs. Morph B) from a short clip of them talking in a two-person conversation. Fine-tuned on Meta's Seamless Interaction dataset.

Code / training pipeline: see the accompanying GitHub repository.

Files

File	Description
`best_acc.ckpt`	PyTorch-Lightning checkpoint (~1 GB), best `val/acc`
`config.yaml`	exact training config for this run
`MODEL_CARD.md`	this card

Model

Base model: MCG-NJU/videomae-base-finetuned-kinetics (~86 M params), loaded via transformers.VideoMAEForVideoClassification. The K400 400-way head is replaced with a 2-way (Morph A / Morph B) head; the backbone is fine-tuned.
Input: 64-frame clips @ 4 fps, short-side 224 → top-cropped to 224×224, ImageNet mean/std normalized. VideoMAE's sinusoidal temporal position embeddings scale from the pretrained 16 frames to 64 with no interpolation.

Training

Data: ~2,462 per-speaker clips (1,970 train / 492 test), classes ~62% A / 38% B.
Split: random per-clip 80/20, stratified by class.
Optimizer: AdamW, head lr 1e-4 / backbone lr 1e-5 (×0.1), weight decay 0.05.
Schedule: 10-epoch warmup + cosine decay to 1e-6; early stopping on val/acc.
Loss: class-weighted cross-entropy.

Results

Metric	Value
Val accuracy	~99.6%
Val F1	~99.5%

Intended use

Research baseline for whether the behavioral "Morph" distinction is decodable from raw video.

Usage

bash

huggingface-cli download QingCheng24/seamless_morph --local-dir weights

python train_videomae.py --test --ckpt_path weights/best_acc.ckpt

License

Released under CC BY-NC 4.0, inheriting the non-commercial terms of the Seamless Interaction dataset and the Morph A/B annotations it was trained on.