BS → AU PLS

Predicts 20 FACS Action Unit intensities from 52 MediaPipe blendshapes via Cheong-style PLS regression. Lets MPDetector output AU columns comparable to Detector's xgb output.

BS -> AU PLS (v2)

Training data

• 350,568 frames from ~10,000 CelebV-HQ celebrity videos
• Paired blendshapes (MPDetector mp_blendshapes head) + AU intensities (Detector with img2pose face + xgb AU on the same frames)
• Pose-filtered to |yaw| <= 40°, |pitch| <= 30° -> 347,897 retained
• 9,994 unique videos after filtering
• See /Storage/Projects/mp_blendshapes for the underlying training pipeline

Method

• PLSRegression(n_components=20, scale=True), Cheong / Py-Feat style
• 20 components = full rank (capped at min(n_features=52, n_targets=20))
• Linear features only — pairwise BS interactions were tested in nested CV (2026-05-05) and HURT out-of-sample R² (bs_only=0.236 vs bs_pairs=0.214, with 4-6x higher fold std)
• No pose covariates: kept pose-agnostic since MP blendshapes are designed to be pose-canonical
• No clipping at training (clip to [0,1] at inference if desired)

Performance (3-fold GroupKFold by video_id)

• Overall R² = 0.236 +/- 0.008 (variance-weighted across 20 AUs)
• Overall MAE = 0.171
• Strong on AU06/12/43 (~0.50)
• Moderate on AU01/02/09 (~0.29)
• Weak on AU11/15/28 (<0.10) — these are rare or visually subtle AUs

Citation context

• Cheong et al. 2023 (Py-Feat AU visualization model, tutorial 06 by E. Jolly): affine-aligned 68 dlib landmarks -> 20 AUs via PLS on EmotioNet/DISFA/BP4D (~13K class-balanced rows). Our model scales the recipe up to MP blendshapes on 10x larger wild-celebrity data.

Inference

au = blendshapes @ coef + intercept   # (n, 52) @ (52, 20) + (20,) = (n, 20)
au = np.clip(au, 0.0, 1.0)            # optional

File format

• coef (52, 20) float32 — linear weights, rows match bs_columns
• intercept (20,) float32 — bias, matches au_columns
• bs_columns (52,) str — input feature order
• au_columns (20,) str — output AU order
• model_card () str — this markdown
• training_metadata () str — JSON dict with training context