In this study, we utilized the free version of FaceForen-
sics++ dataset and the Celeb-DF dataset available
on Kaggle. From these datasets, we selected a total of 1,580
videos, with an equal distribution of 790 real and 790 fake
videos.
To prepare the data for training, we first calculated the
duration of each video to ensure consistent frame extraction.
We then extracted 20 evenly spaced frames from each video,
allowing us to capture representative visual information while
avoiding redundancy. For each extracted frame, we employed
the MTCNN (Multi-task Cascaded Convolutional Networks)
to detect and crop the faces. MTCNN is a popular face
detection algorithm that efficiently localizes facial regions
while handling variations in pose and lighting. Each
detected face was resized to a uniform size of 224x224 pixels
to maintain consistency across the dataset. This preprocessing
pipeline helped in standardizing the input data, making it
suitable for training the deep learning model. The cropped
face images were saved for subsequent training.
During training, we applied data augmentation techniques
to increase the diversity and robustness of the model.
These augmentations simulate real-world variations, helping
the model generalize better. The augmentations included
random resized cropping, horizontal flipping, rotation, color
jittering, Gaussian blurring, affine transformations, grayscale
conversion, and normalization. These transformations mimic
different visual conditions, thereby reducing the risk of over-
fitting.
Model Architecture and Methodology
Model Architecture
Deep learning models have shown remarkable performance
in the field of image analysis and classification. CNNs have
traditionally been the go-to architecture due to their ability
to capture localized spatial features effectively. How-
ever, when it comes to understanding complex patterns and
long-range dependencies in images, ViTs have emerged as
a more powerful alternative. In this study, we leverage
the strengths of both architectures by combining EfficientNet-
B0 for spatial feature extraction with a ViT for relational
modeling. This hybrid approach aims to efficiently capture
both fine-grained and global patterns inherent in deepfake
videos.
The core of our model architecture is a hybrid design
that integrates EfficientNet-B0 and a Vision Transformer
(ViT). EfficientNet-B0 acts as the feature extractor, providing
spatial representations, while the ViT component captures
long-range dependencies through self-attention.
The input to the model is a batch of face images. These
images are first processed through EfficientNet-B0, which
extracts spatial features in the form of 1,280 feature maps,
each of size 7x7, resulting in a tensor of shape [batch, 1280,
7, 7]. To make these features suitable for the transformer
encoder, we flatten the spatial dimensions, transforming the
tensor to [batch, 49, 1280], where 49 represents the number of
tokens derived from the flattened feature maps. This structure
can be understood by drawing a parallel to natural language
processing (NLP), where each token represents a word.
Here, each of the 49 tokens corresponds to a spatial patch of
the image.
Next, we reduce the dimensionality of each feature vector
from 1,280 to 384 using a linear layer. This results in a tensor
of shape [batch, 49, 384]. A learnable CLS (classification)
token is appended to the beginning of the sequence, forming
a tensor of shape [batch, 50, 384]. This token aggregates
information from the entire image and is ultimately used
for classification. Positional embeddings are then added to
retain spatial information since transformers inherently lack
the ability to capture positional relationships.
The resulting tensor is passed through six consecutive ViT
blocks. Each block comprises multi-head attention (MHA) and
feed-forward neural network layers. The MHA mechanism
allows the model to focus on various parts of the image
simultaneously, capturing complex interdependencies between
facial features. Residual connections are incorporated
within each ViT block to enhance gradient flow, preventing
the vanishing gradient problem. Dropout layers are added at
various stages to reduce overfitting and improve generalization.
The final representation from the CLS token is passed
through an MLP head comprising linear and GELU layers
to output the probability of the input being a deepfake. This
combination of EfficientNet for spatial encoding and ViT for
relational modeling leads to a robust architecture capable of
capturing both local and global patterns.
The loss function used for training is Binary Cross En-
tropy with Logits (BCEWithLogitsLoss), which is well-
suited for binary classification tasks as it combines a sigmoid
layer with binary cross-entropy loss. The optimizer chosen
is AdamW, known for its effective weight decay, with
a learning rate of 1e-4 and weight decay of 1e-4. To further
optimize training, we use a learning rate scheduler that reduces
the learning rate by a factor of 0.5 when the validation loss
plateaus for three epochs, promoting efficient convergence.
Inference Pipeline
Inference Pipeline
Comparison of model performance with and without Transformer on the FaceForensics++ and Caleb DF datasets
Screenshot 2026-06-01 191550
Performance of the proposed model at different batch sizes
Screenshot 2026-06-01 191607
This repository contains a PyTorch model for deepfake detection based on an improved EfficientViT architecture, trained on video data.
The model predicts whether a video is real (0) or fake (1) using both visual information and temporal cues.
🧩 Model Description
Architecture: Improved EfficientViT Backbone: EfficientNet-B0 for feature extraction Head: Transformer-based temporal modeling with classification head Input: Video frames (224×224 RGB images) Output: Binary label (0=Real, 1=Fake) and frame-level probabilities
Key Features:
Extracts faces from frames using MTCNN
Supports inference on raw video files
Provides frame-level probabilities for fine-grained analysis
📁 Repository Structure
deepfake-efficientvit/
│
├── model.py # ImprovedEfficientViT class
├── inference.py # Functions to run inference on videos
├── model.pth # Trained weights
├── config.json # Optional model metadata
├── requirements.txt # Required packages
├── README.md
