🔍 Convolutional Vision Transformer (CViT) for Deepfake Detection

Model Architecture

1. Feature Learning (FL) Module - CNN Backbone

• Composed of 17 convolutional operations.
• Unlike traditional VGG architectures, FL focuses purely on feature extraction, not classification.
• Accepts input of size 224 × 224 × 3 (RGB image).
• Outputs a 512 × 7 × 7 feature map.
• Contains 10.8 million learnable parameters.

2. Vision Transformer (ViT) Module

• Receives CNN output (512 × 7 × 7) as its input.
• Converts the 7×7 patches into a 1 × 1024 sequence using linear embedding.
• Adds positional embeddings of shape (2 × 1024).
• ViT Encoder uses:
  • Multi-Head Self Attention (MSA) with 8 attention heads.
  • MLP blocks with:
    • First linear layer of 2048 units.
    • Final linear layer of 2 units (binary classification: Fake / Real).
    • ReLU activation and Softmax for final probabilities.

🧪 Experimental Results

📊 FaceForensics++ Accuracy

Note: Poor performance on the FaceShifter dataset is attributed to the model's difficulty in learning subtle visual artifacts.

🧪 DFDC Evaluation

• Unseen DFDC test videos: 400
• Accuracy: 91.5%
• AUC Score: 0.91

🧪 UADFV AUC Comparison

⚙️ Training Configuration

• Loss Function: Binary Cross Entropy (BCE)
• Optimizer: Adam
• Learning Rate: 1e-4
• Weight Decay: 1e-6
• Batch Size: 32
• Epochs: 50
• Learning Rate Scheduler: Reduces LR by factor of 0.1 every 15 epochs
• Normalization:
  • Mean: [0.485, 0.456, 0.406]
  • Std: [0.229, 0.224, 0.225]

🧪 Inference Setup

• Input: 30 normalized facial images (per video)
• Classification:
  • Uses log loss function to compute confidence.
  • Output is a probability y ∈ [0, 1]
    • 0 < y < 0.5: Real
    • 0.5 ≤ y ≤ 1: Fake
• Log loss penalizes:
  • Random guesses
  • Confident but incorrect predictions

🛠 Inference Example