Anime Frame Interesting Classifier (ViT v2.0)
Model Details
Architecture: MobileViT-Small (Transformer)
Framework: Hugging Face Transformers
Input Size: 224x224 RGB images
Output: Binary classification (Boring/Interesting)
Performance
Evaluated on v2.0 test set (433 frames):
- F1 Score: 94.92%
- Accuracy: 94.92%
- Precision: 95.05%
- Recall: 94.92%
Training data: 3,999 frames (4,432 total with test holdout)
Intended Use
What it does: Classifies anime frames as either "interesting" (depicting meaningful character/scene details) or "boring" (back-of-head shots, non-descript backgrounds, montages).
Strengths:
- Transformer-based semantic understanding
- Better generalization to style variations
- Good for ensemble voting with CNN model
- Complementary confidence to CNN predictions
When to use:
- Ensemble voting with CNN model for higher confidence
- Applications preferring transformer-based features
- Fine-tuning for downstream anime tasks
When NOT to use:
- Real-world photos or non-anime content
- Frames smaller than 224x224
- Speed-critical deployments (slower than CNN)
Labels
- Class 0 (Boring): Frames lacking interesting visual details or character focus
- Class 1 (Interesting): Frames with clear character/scene details suitable for downstream tasks
Model Size & Speed
- Model Size: 19 MB (SafeTensors format)
- Inference Speed: ~25ms per image on GPU
- VRAM Required: ~2 GB (including activations)
- Speed vs CNN: ~25% slower but more semantically aware
Training Data Composition
- 900 manually curated frames (hand-labeled)
- 1,655 frames filtered via dual-agreement with garbage classifier
- 1,877 frames from curated anime site scraper
- Total: 4,432 frames (90% train, 10% test holdout)
All frames are 224x224 RGB anime screenshots.
How to Use
Recommended: HuggingFace Transformers (SafeTensors)
1from transformers import AutoImageProcessor, AutoModelForImageClassification
2from PIL import Image
3
4# Load model (automatically uses SafeTensors)
5processor = AutoImageProcessor.from_pretrained(
6 'hf_models/anime-frame-interesting-classifier-vit-v2'
7)
8model = AutoModelForImageClassification.from_pretrained(
9 'hf_models/anime-frame-interesting-classifier-vit-v2',
10 trust_remote_code=False # Safe: SafeTensors prevents code execution
11)
12model.eval()
13
14image = Image.open('frame.png')
15inputs = processor(image, return_tensors="pt")
16
17with torch.no_grad():
18 outputs = model(**inputs)
19 logits = outputs.logits
20 prediction = logits.argmax(-1).item()
21 confidence = logits.softmax(-1)[0][prediction].item()
22
23print(f"Prediction: {'Interesting' if prediction == 1 else 'Boring'}")
24print(f"Confidence: {confidence:.2%}")
Direct Load with SafeTensors
1from transformers import MobileViTForImageClassification, AutoImageProcessor
2from safetensors.torch import load_file
3from PIL import Image
4import torch
5
6# Load with SafeTensors (secure)
7model = MobileViTForImageClassification.from_pretrained(
8 'apple/mobilevit-small',
9 num_labels=2
10)
11state_dict = load_file('model.safetensors')
12model.load_state_dict(state_dict)
13model.eval()
14
15processor = AutoImageProcessor.from_pretrained('apple/mobilevit-small')
16image = Image.open('frame.png')
17inputs = processor(image, return_tensors="pt")
18
19with torch.no_grad():
20 outputs = model(**inputs)
21 prediction = outputs.logits.argmax(-1).item()
22
23print(f"Prediction: {'Interesting' if prediction == 1 else 'Boring'}")
Security Note: This model uses SafeTensors format (not pickle). SafeTensors is a secure serialization format that cannot execute arbitrary code during loading, unlike pickle-based .bin files.
Comparison with CNN Model
See anime-frame-interesting-classifier-cnn-v2 for CNN alternative:
| Metric | ViT (This Model) | CNN | Use Case |
|---|
| F1 | 94.92% | 95.15% | ViT: ensemble, CNN: general |
| Speed | Slower | Faster | CNN preferred for speed |
| Size | 20 MB | 20 MB | Similar footprint |
| Semantics | Better understanding | Good efficiency | ViT for understanding |
| Ensemble | Better recall with CNN voting | Better precision with ViT voting | Use together |
Recommended: Use ensemble voting for maximum confidence:
- Classify with both models
- Flag disagreements for manual review
- Trust when both models agree
Limitations
- Anime-only: Trained exclusively on anime content
- Speed: Slower inference than CNN (1-2 fps vs 2-3 fps)
- Dataset bias: Training data skewed toward popular anime styles
- Resolution: Trained on 224x224; extreme aspect ratios need preprocessing
- Edge cases: Minimal training on hard-to-classify borderline frames
Training Details
- Dataset: v2.0 (4,432 frames)
- Base Model: apple/mobilevit-small (5.6M parameters)
- Train/Test Split: 90/10 (3,999 train, 433 test)
- Epochs: 20
- Batch Size: 64
- Optimizer: AdamW (lr=1e-4)
- Loss: CrossEntropyLoss
- Augmentation: None (data quality sufficient at this scale)
Version History
- v2.0 (current): 94.92% F1, retrained on expanded 4,432-frame dataset
- v1.0: 86% F1, 900-frame dataset (legacy, deprecated)
Citation
If you use this model, please reference:
- Dataset: Anime Frame Interesting v2.0 (4,432 curated frames)
- Architecture: MobileViT-Small (apple/mobilevit-small)
- Framework: Hugging Face Transformers
- Training: PyTorch, 2026
Ensemble Strategy
For best results, use both CNN and ViT models together:
1def ensemble_classify(image_path, cnn_model, vit_model):
2 """Classify with both models, flag disagreements"""
3 # CNN prediction
4 cnn_pred = classify_with_cnn(image_path, cnn_model)
5
6 # ViT prediction
7 vit_pred = classify_with_vit(image_path, vit_model)
8
9 if cnn_pred['class'] == vit_pred['class']:
10 # Agreement: high confidence
11 confidence = (cnn_pred['conf'] + vit_pred['conf']) / 2
12 return {
13 'prediction': cnn_pred['class'],
14 'confidence': confidence,
15 'agreement': 'both'
16 }
17 else:
18 # Disagreement: flag for manual review
19 return {
20 'cnn': cnn_pred,
21 'vit': vit_pred,
22 'agreement': 'none',
23 'recommendation': 'manual_review'
24 }
Future Improvements
- Collect 1,000+ edge-case frames for hard-negatives
- Experiment with larger ViT variants (if available)
- Fine-tune for specific anime styles
- Distill to smaller model for faster inference
