Architecture: CLIPForImageClassification (CLIP ViT-B/32 vision encoder + linear classification head)
Base Model: openai/clip-vit-base-patch32
Training Data: PlantVillage Dataset (54K+ labeled leaf images)
Classes: 38 (26 diseases + 12 healthy states across 14 crop species)
Training Recipe: Based on "CLIP Itself is a Strong Fine-tuner"

Usage

python

from transformers import CLIPForImageClassification, AutoImageProcessor
from PIL import Image
import torch

model = CLIPForImageClassification.from_pretrained("VaigandlaHemanth/leaf-disease-clip-vit")
processor = AutoImageProcessor.from_pretrained("VaigandlaHemanth/leaf-disease-clip-vit")

image = Image.open("leaf.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

predicted_class = outputs.logits.argmax(-1).item()
label = model.config.id2label[str(predicted_class)]
print(f"Predicted: {label}")

Supported Classes (38)

Plant	Diseases
Apple	Scab, Black Rot, Cedar Apple Rust, Healthy
Blueberry	Healthy
Cherry	Powdery Mildew, Healthy
Corn	Gray Leaf Spot, Common Rust, Northern Leaf Blight, Healthy
Grape	Black Rot, Esca, Leaf Blight, Healthy
Orange	Huanglongbing (Citrus Greening)
Peach	Bacterial Spot, Healthy
Pepper	Bacterial Spot, Healthy
Potato	Early Blight, Late Blight, Healthy
Raspberry	Healthy
Soybean	Healthy
Squash	Powdery Mildew
Strawberry	Leaf Scorch, Healthy
Tomato	Bacterial Spot, Early/Late Blight, Leaf Mold, Septoria, Spider Mites, Target Spot, TYLCV, Mosaic Virus, Healthy

Fine-Tuning (GPU)

A complete training script is included in train.py. To fine-tune on GPU:

bash

pip install transformers datasets torch torchvision scikit-learn accelerate
python train.py

Key hyperparameters (from paper):

Learning Rate: 2e-5 (critical — CLIP needs ~50x smaller LR than standard ViT)
Label Smoothing: 0.1
Weight Decay: 0.05
Schedule: Cosine with 10% warmup
Augmentation: RandomResizedCrop, HorizontalFlip, Rotation, ColorJitter (NO MixUp/CutMix)

Expected accuracy with full GPU training: >95% on PlantVillage test set.

Demo

Try the interactive demo: 🌿 Leaf Disease Detector Space

Citation

bibtex

@article{dong2022clip,
  title={CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet},
  author={Dong, Xiaoyi and Bao, Jianmin and Zhang, Ting and Chen, Dongdong and Zhang, Weiming and Yuan, Lu and Chen, Dong and Wen, Fang and Yu, Nenghai},
  journal={arXiv preprint arXiv:2212.06138},
  year={2022}
}