vit-trm-ssv2 – AI Model by bcgxtberg | AlphaNeural AI

You can deploy this model and start earning money today!

bcgxtberg/
vit-trm-ssv2

like

0

video-classification

action-recognition

recursive-reasoning

pytorch-lightning

flwrlabs/ucf101

adelabdalla221/vit-trm-hmdb51

Views

No views yet

ViT-TRM on UCF-101

Extends the ViT-TRM architecture from HMDB51 (51 classes) to UCF-101 (101 action classes) via transfer learning.

Architecture

Video Frames → ViT (per-frame) → Mean Pool → Positional Encoding
    → TRM Reasoning (H=2 cycles, L=2 shared layers) → Mean Pool → Classifier (101 classes)

Setup

pip install torch torchvision pytorch-lightning timm torchmetrics datasets

Training

bash

# Transfer from HMDB51 (recommended)
python train_ucf101.py --pretrained_ckpt vit-trm-epoch=29-val_acc=0.7113.ckpt

# From scratch
python train_ucf101.py

# Smoke test
python train_ucf101.py --fast_dev_run --max_videos 50

Usage

python

import torch
from vit_trm_video import ViTTRMVideo

model = ViTTRMVideo.load_from_checkpoint("checkpoints/best.ckpt", strict=False)
model.eval()

video = torch.randn(1, 16, 3, 224, 224)  # (batch, frames, C, H, W)
with torch.no_grad():
    logits = model(video)
    pred = logits.argmax(dim=-1)