You can deploy this model and start earning money today!

naifenn/
motivate-pose-encoder

Name: motivate-pose-encoder
Availability: InStock

tensorboard

Views

No views yet

Pose Encoder

Trains a shared adapter on top of a frozen PoseFormerV2 encoder for two tasks:

Metric rating — Good / Okay / Needs work per exercise metric
Exercise classification — which exercise is being performed

The trained adapter weights feed into the multimodal MotiVate pipeline.

Pipeline

MediaPipe 2D
→ H36M remap + pad/crop + normalize
→ PoseFormerV2 (frozen, 27×544)
→ Shared Adapter (trainable, 27×256)
→ mean pool
→ Rating Head + Exercise Head

Loss = rating_loss + exercise_loss_weight × exercise_loss

Data Sources

Source	Path	Used for
`training.csv` / `validation.csv`	`train/unimodal/`	Train/val split only — provides `(dataset, clip_id)` pairs. No labels are read from these CSVs.
`pose_data.npz`	`processed/<dataset>/<clip_id>/mediapipe_result/`	Model input — MediaPipe 2D pose landmarks per frame.
`raw_gt.csv`	`processed/<dataset>/<clip_id>/pose/`	Rating labels — raw pose measurements. Fed to `compute_metrics()` at load time to produce `Good` / `Okay` / `Needs work` per metric.
`metadata.json`	`processed/<dataset>/`	Exercise labels — maps each `clip_id` to its exercise name (fallback: infer from `clip_id` prefix).

Labels are not pre-computed in the CSVs. They are derived on the fly:

Rating targets: raw_gt.csv → compute_metrics(df, exercise) → evaluate_rating() → one of Good(0) / Okay(1) / Needs work(2) per metric. Metrics that can't be evaluated get IGNORE_INDEX = -100 and are excluded from the loss.
Exercise targets: looked up from metadata.json and mapped to a class index via EXERCISE_TO_ID.

Setup

bash

cd MotiVate/train
bash setup_poseformer.sh          # clone + download checkpoint
bash setup_poseformer.sh --force  # re-clone

Training

uv run python train/pose_encoder/train_shared_adapter.py --config train/pose_encoder/config.json

Outputs

Saved to checkpoints/shared_adapter/:

best_shared_adapter.pt — adapter weights for the multimodal pipeline
best_model.pt / last_model.pt — full checkpoints

val_score = 0.5 × (rating_acc + exercise_acc) — used for scheduling, early stopping, and best checkpoint selection.

naifenn/motivate-pose-encoder