🎬 RetFormer: Efficient TimeSformer + RetNet for Video Action Recognition
RetFormer is a hybrid video classification model that replaces the temporal attention in TimeSformer with RetNet, achieving:
- ⚡ Lower memory usage
- 🚀 Faster training
- 🎯 Competitive accuracy
🧠 Model Architecture
🔹 RetFormer (Proposed)
- Spatial Modeling → TimeSformer
- Temporal Modeling → RetNet
👉 This replaces quadratic attention with linear-time temporal modeling (O(n))
📊 Dataset
- HMDB51
- 51 human action classes
- Complex motion patterns
- Smaller and more challenging than UCF101
🔁 Training Strategy
Training was performed in multiple stages due to runtime limits:
- Initial training (Epoch 1–10)
- Checkpoint saving
- Resumed training (Epoch 11–14)
- Early stopping applied
📈 Training Results (Epoch 1–14)
| Epoch | Train Loss | Train Acc | Val Loss | Val Acc | F1 |
|---|
| 1 | 3.9312 | 0.0350 | 3.8099 | 0.0967 | 0.0855 |
| 2 | 3.6330 | 0.1791 | 3.2948 | 0.3654 | 0.3149 |
| 3 | 3.0989 | 0.3691 | 2.6927 | 0.5150 | 0.4579 |
| 4 | 2.6278 | 0.5048 | 2.2879 | 0.5869 | 0.5503 |
| 5 | 2.3198 | 0.5782 | 2.0438 | 0.6255 | 0.5961 |
| 6 | 2.1387 | 0.6194 | 1.9152 | 0.6242 | 0.6074 |
| 7 | 1.9876 | 0.6657 | 1.8369 | 0.6418 | 0.6308 |
| 8 | 1.9140 | 0.6936 | 1.7966 | 0.6359 | 0.6188 |
| 9 | 1.8539 | 0.7041 | 1.7619 | 0.6556 | 0.6426 |
| 10 | 1.8149 | 0.7244 | 1.7523 | 0.6614 | 0.6512 |
| 11 | 1.7325 | 0.7524 | 1.7315 | 0.6699 | 0.6614 |
| 12 | 1.7036 | 0.7584 | 1.7469 | 0.6621 | 0.6515 |
| 13 | 1.6682 | 0.7717 | 1.7504 | 0.6595 | 0.6496 |
| 14 | 1.6344 | 0.7785 | 1.7488 | 0.6588 | 0.6494 |
🏆 Best Performance
- Validation Accuracy: 66.99%
- F1 Score: 0.6614
- Achieved at Epoch 11
⚙️ Training Details
- Peak GPU Memory: ~7.2 GB
- Training Time per Epoch: ~52 minutes
- Evaluation Time: ~8 minutes
- Mixed Precision Training (
torch.cuda.amp)
- Early stopping triggered after Epoch 14
📌 Observations
- Stable improvement until Epoch 11
- Slight decline afterward → early overfitting
- Lower accuracy than baseline (expected for hybrid trade-off)
⚡ Efficiency Advantage
| Metric | TimeSformer | RetFormer |
|---|
| Peak GPU Memory | ~9.3 GB | ~7.2 GB ✅ |
| Complexity | O(n²) | O(n) ✅ |
| Speed | Slower | Faster |
👉 ~25% reduction in GPU memory
🔍 Key Insight
RetFormer demonstrates that:
- Efficient temporal modeling can significantly reduce memory usage
- Performance remains competitive with baseline models
- Trade-off exists between efficiency and maximum accuracy
🚀 Usage
pip install torch torchvision transformers
