PlantDiagRAG combines disease classification, visual question answering (VQA), and RAG-based treatment recommendations into a single unified system.

Model Description

PlantDiagRAG is a multi-task vision-language model for plant disease diagnosis:

Classification: Identifies 38 plant disease classes with 99.10% accuracy
VQA: Answers natural language questions about plant diseases
RAG: Retrieves treatment recommendations from agricultural knowledge bases

Architecture

ViT-Base (Vision) + BERT-Base (Text) → Cross-Modal Attention → Flan-T5-Base (LoRA)
                                                             ↓
                                            ┌────────────────┼────────────────┐
                                            ↓                ↓                ↓
                                      Classification       VQA            RAG Retrieval

Component	Model	Parameters
Vision Encoder	google/vit-base-patch16-224	86M
Text Encoder	bert-base-uncased	110M
Language Model	google/flan-t5-base + LoRA	248M
Total		449M (4.5% trainable)

Performance

Classification (38 classes)

Metric	Score
Accuracy	99.10%
Precision	99.16%
Recall	99.11%
F1-Score	99.11%

Visual Question Answering

Metric	Score
BLEU	21.63%
ROUGE-1	64.63%
ROUGE-L	63.38%
METEOR	44.05%
Token F1	67.82%

Inference Speed (NVIDIA A100)

Task	Latency
Classification	24ms
VQA	156ms

Files

File	Size	Description
`best_vqa_model.pt`	~1.76 GB	VQA model (ViT + BERT + T5-Base + LoRA)
`best_classifier_v2.pt`	~1.06 GB	Classification model (ViT + BERT)
`all_kb_documents.json`	~500 KB	RAG knowledge base (54 documents)

Usage

Installation

bash

pip install torch torchvision transformers==4.36.0 peft==0.7.0
pip install sentence-transformers faiss-cpu

Quick Start

python

from huggingface_hub import hf_hub_download
import torch

# Download model
checkpoint_path = hf_hub_download(
    repo_id="vivek-sakhiya/PlantDiagRAG",
    filename="best_vqa_model.pt"
)

# Load checkpoint
checkpoint = torch.load(checkpoint_path, map_location="cuda")
print(f"Epoch: {checkpoint['epoch']}, Val Loss: {checkpoint['val_loss']:.4f}")

Full Pipeline

See the GitHub repository for complete usage with the PlantDiagRAGPipeline class.

Supported Disease Classes

Click to expand all 38 classes

#	Class	Plant	Disease
0	Apple___Apple_scab	Apple	Apple Scab
1	Apple___Black_rot	Apple	Black Rot
2	Apple___Cedar_apple_rust	Apple	Cedar Apple Rust
3	Apple___healthy	Apple	Healthy
4	Blueberry___healthy	Blueberry	Healthy
5	Cherry_(including_sour)___Powdery_mildew	Cherry	Powdery Mildew
6	Cherry_(including_sour)___healthy	Cherry	Healthy
7	Corn_(maize)___Cercospora_leaf_spot_Gray_leaf_spot	Corn	Gray Leaf Spot
8	Corn_(maize)__Common_rust	Corn	Common Rust
9	Corn_(maize)___Northern_Leaf_Blight	Corn	Northern Leaf Blight
10	Corn_(maize)___healthy	Corn	Healthy
11	Grape___Black_rot	Grape	Black Rot
12	Grape___Esca_(Black_Measles)	Grape	Esca
13	Grape___Leaf_blight_(Isariopsis_Leaf_Spot)	Grape	Leaf Blight
14	Grape___healthy	Grape	Healthy
15	Orange___Haunglongbing_(Citrus_greening)	Orange	Citrus Greening
16	Peach___Bacterial_spot	Peach	Bacterial Spot
17	Peach___healthy	Peach	Healthy
18	Pepper,_bell___Bacterial_spot	Pepper	Bacterial Spot
19	Pepper,_bell___healthy	Pepper	Healthy
20	Potato___Early_blight	Potato	Early Blight
21	Potato___Late_blight	Potato	Late Blight
22	Potato___healthy	Potato	Healthy
23	Raspberry___healthy	Raspberry	Healthy
24	Soybean___healthy	Soybean	Healthy
25	Squash___Powdery_mildew	Squash	Powdery Mildew
26	Strawberry___Leaf_scorch	Strawberry	Leaf Scorch
27	Strawberry___healthy	Strawberry	Healthy
28	Tomato___Bacterial_spot	Tomato	Bacterial Spot
29	Tomato___Early_blight	Tomato	Early Blight
30	Tomato___Late_blight	Tomato	Late Blight
31	Tomato___Leaf_Mold	Tomato	Leaf Mold
32	Tomato___Septoria_leaf_spot	Tomato	Septoria Leaf Spot
33	Tomato___Spider_mites_Two-spotted_spider_mite	Tomato	Spider Mites
34	Tomato___Target_Spot	Tomato	Target Spot
35	Tomato___Tomato_Yellow_Leaf_Curl_Virus	Tomato	Yellow Leaf Curl Virus
36	Tomato___Tomato_mosaic_virus	Tomato	Mosaic Virus
37	Tomato___healthy	Tomato	Healthy

Training Data

PlantVillage: 54,306 images across 38 classes
PlantVillageVQA: 193,609 question-answer pairs

Knowledge Base

RAG retrieval uses 54 documents from authoritative agricultural sources:

Source	Documents	Percentage
ICAR (Indian Council of Agricultural Research)	18	33.3%
UC IPM (UC Integrated Pest Management)	15	27.8%
PNW Handbook (Pacific Northwest Plant Disease)	12	22.2%
AGROVOC (FAO Agricultural Vocabulary)	9	16.7%

Author

Vivekkumar Sakhiya
M.Tech in AI & Data Science, MNNIT Allahabad
Supervisor: Dr. Abhinav Kumar

Made with 🌱 for sustainable agriculture

vivek-sakhiya/PlantDiagRAG

🌱 PlantDiagRAG