This repository provides a Transformer + BiLSTM-based Emotion Recognition model trained on MediaPipe landmark sequences extracted from facial points. The model classifies human emotions into six categories: Angry, Disgust, Fear, Happy, Neutral, Sad.
It processes temporal sequences of 256 frames per clip with 478 landmarks per frame, learning the dynamic patterns of human expression. The model is optimized for real-time emotion inference and can be used in applications such as sign language understanding and emotion-aware human-computer interaction.
🧩 Model Architecture
The model is built using Transformer layers for sequence modeling:
Input Layer:
Accepts sequences of shape (256, 478*3) corresponding to 3D coordinates of 478 landmarks over 256 frames.
Transformer Encoder Layers:
Capture temporal dependencies and dynamic patterns of human motion using self-attention mechanisms.
Fully Connected Layers:
Transform the encoder outputs into probabilities for six emotion classes.
Output Layer:
Softmax activation for multi-class emotion classification.
📊 Dataset
Custom MediaPipe Landmark Dataset
Extracted from labeled video clips representing six emotions.
Preprocessing includes normalization, sequence grouping (256 frames per clip), and balanced augmentation.
Total dataset is split into training, validation, and test sets.
To correctly use this model for prediction, you must first preprocess your video data using the provided assets for standardization and label encoding.
1. Preprocessing Assets
The necessary files for video preprocessing are stored in the assets/ folder of this repository:
File Name
Purpose
Required for Step
emotion_label_encoder.joblib
Maps predicted indices back to human-readable emotion labels (e.g., 0 -> 'Happy').
Post-Inference
global_mean_tensor.pt
Global mean tensor used to normalize the extracted MediaPipe features.
Preprocessing
global_std_tensor.pt
Global standard deviation tensor used to normalize the extracted MediaPipe features.
Preprocessing
You must load the mean tensor and std tensor to standardize your input feature sequences before feeding them into the BiLSTM model.
2. Complete Example
For a full, runnable demonstration showing how to load the model, use the assets for standardization, and run inference on a video, please refer to the usage notebook:
