AgriVision BLIP2
AgriVision BLIP2 is a fine-tuned multimodal Vision-Language Model developed for intelligent crop disease diagnosis from plant leaf images.
Built on top of Salesforce BLIP2 OPT 2.7B and adapted using LoRA (Low-Rank Adaptation), the model generates structured agricultural diagnosis reports containing disease identification, symptom interpretation, pathogen categorization, and agronomic recommendations.
Unlike traditional image classification models that only predict disease labels, AgriVision BLIP2 is designed to provide explainable and human-readable diagnostic outputs.
Model Overview
Objective
The primary objective of this project is to transform a general-purpose Vision-Language Model into a domain-specialized agricultural assistant capable of understanding crop diseases and generating structured diagnostic reports.
The model performs multimodal reasoning by jointly analyzing visual leaf patterns and generating natural language agricultural explanations.
Key Capabilities
The model can generate:
- Crop disease identification
- Symptom descriptions
- Pathogen category analysis
- Structured agricultural diagnosis reports
- Clinical-style disease explanations
Example output:
This is a diseased Maize leaf. Disease identified: Northern Leaf Blight. Visible symptoms include canoe-shaped lesions with gray-green margins that turn tan with dark fungal sporulation, starting on lower leaves and spreading upwards. Pathogen category: Fungal.
Model Details
| Attribute | Value |
|---|
| Model Name | AgriVision BLIP2 |
| Base Model | Salesforce/blip2-opt-2.7b |
| Architecture | Vision-Language Model (VLM) |
| Fine-Tuning Method | LoRA |
| Framework | Transformers + PEFT + PyTorch |
| Domain | Agricultural Disease Diagnosis |
| Language | English |
| License | Apache 2.0 |
Dataset
LeafNet
This model was fine-tuned using the LeafNet dataset.
Dataset:
The dataset contains:
- Multiple crop species
- Healthy and diseased leaves
- Fungal diseases
- Bacterial diseases
- Pest and insect damage
- Symptom-rich annotations
Training samples were converted into multiple instruction-style agricultural reasoning tasks.
Training Strategy
Fine-Tuning Approach
The model was adapted using:
- LoRA (Low-Rank Adaptation)
- 8-bit Quantization
- Mixed Precision Training
- Multimodal Agricultural Instruction Tuning
This approach enabled efficient domain adaptation while preserving the general multimodal capabilities of BLIP2.
Training Tasks
The model was trained on multiple agricultural reasoning tasks:
Disease Identification
Example:
Input:
Plant leaf image
Output:
This is a Potato leaf affected by Early blight.
Symptom Explanation
Example:
Input:
Plant leaf image
Output:
Small, dark, papery flecks growing into brown-black circular lesions.
Pathogen Classification
Example:
Input:
Plant leaf image
Output:
This plant shows a Fungal condition.
Comprehensive Agricultural Diagnosis
Example:
Input:
Plant leaf image
Output:
This is a diseased Coffee leaf. Disease identified: Phoma. Visible symptoms include dark-colored zoned patches starting at the edges with small black pycnidia on the lesions. Pathogen category: Fungal.
Intended Use
Direct Use
The model is intended for:
- Agricultural disease diagnosis
- Plant pathology research
- Agritech applications
- Educational demonstrations
- AI-assisted crop monitoring systems
Downstream Applications
Potential downstream use cases include:
- Smart farming platforms
- Mobile crop diagnosis applications
- Precision agriculture systems
- Agricultural advisory tools
- Farmer support systems
Limitations
Users should be aware of the following limitations:
- Performance depends on image quality.
- The model may struggle with crop species not represented in the training data.
- Predictions should not replace professional agronomic advice.
- Generated outputs may occasionally contain hallucinated information.
- The model should be treated as a decision-support system rather than a definitive diagnosis tool.
Example Usage
1from PIL import Image
2from transformers import Blip2Processor, Blip2ForConditionalGeneration
3
4processor = Blip2Processor.from_pretrained("YOUR_USERNAME/AgriVision-BLIP2")
5model = Blip2ForConditionalGeneration.from_pretrained("YOUR_USERNAME/AgriVision-BLIP2")
6
7image = Image.open("leaf.jpg")
8
9inputs = processor(
10 images=image,
11 text="Analyze this crop leaf comprehensively.",
12 return_tensors="pt"
13)
14
15outputs = model.generate(
16 **inputs,
17 max_new_tokens=80
18)
19
20print(
21 processor.tokenizer.decode(
22 outputs[0],
23 skip_special_tokens=True
24 )
25)
Future Work
Planned improvements include:
- Larger agricultural datasets
- Better confidence estimation
- Disease-specific remedy generation
- Multilingual support
- Mobile deployment
- Advanced explainability techniques
- Hugging Face Spaces integration
Project Components
The complete AgriVision ecosystem includes:
- Fine-tuned BLIP2 model
- Structured diagnosis engine
- Symptom extraction pipeline
- Remedy recommendation layer
- Explainability heatmaps
- Interactive Gradio application
Acknowledgements
This work builds upon:
- Salesforce BLIP2
- Hugging Face Transformers
- PEFT (Parameter-Efficient Fine-Tuning)
- LeafNet Dataset
- PyTorch
Author
Anhad Mahajan
Computer Science Engineering (Artificial Intelligence)
Interests:
- Multimodal AI
- Computer Vision
- Generative AI
- MLOps
- Agricultural AI
Citation
If you use this model in research or applications, please cite:
1@misc{mahajan2026agrivision,
2 title={AgriVision BLIP2: Intelligent Crop Disease Diagnosis using Vision-Language Models},
3 author={Anhad Mahajan},
4 year={2026},
5 publisher={Hugging Face}
6}
