You can deploy this model and start earning money today!

boudiafA/
AgriChat

Name: AgriChat
Rating: 2.5 (91 reviews)

llava-hf/llava-onevision-qwen2-7b-ov-hf

adapter

apache-2.0

Views

No views yet

AgriChat

Paper

GitHub

AgriChat is a domain-specialized multimodal large language model for agricultural image understanding. It is built on top of LLaVA-OneVision / Qwen-2-7B and adapted with LoRA for fine-grained plant species identification, plant disease diagnosis, and crop counting.

This repository hosts:

the AgriChat LoRA adapter at the repository root
the AgriMM train/test annotation splits under dataset/

Overview

General-purpose MLLMs lack verified agricultural expertise across diverse taxonomies, diseases, and counting settings. AgriChat is trained to address that gap using AgriMM, a large multi-source agricultural instruction dataset covering:

fine-grained plant identification
disease classification and diagnosis
crop counting and grounded visual reasoning

The AgriMM data generation pipeline combines:

image-grounded captioning with Gemma 3 (12B)
verified knowledge retrieval with Gemini 3 Pro and Google Search grounding
QA synthesis with LLaMA 3.1-8B-Instruct

Repository Contents

text

.
├── README.md
├── adapter_config.json
├── adapter_model.safetensors
└── dataset/
    ├── README.md
    ├── train.jsonl
    └── test.jsonl

Model

Base model: llava-hf/llava-onevision-qwen2-7b-ov-hf
Adaptation: LoRA on both the SigLIP vision encoder and the Qwen2 language model
Domain: Agriculture
Main use cases: species recognition, disease reasoning, cultivation-related visual QA, crop counting

Dataset Release

The dataset/ folder contains annotation splits only:

dataset/train.jsonl
dataset/test.jsonl

The repository does not include the source images. Each JSONL line contains an image path relative to a user-created datasets_sorted/ directory. For example:

json

{
  "images": ["datasets_sorted\\iNatAg_subset\\hymenaea_courbaril\\280829227.jpg"],
  "messages": [...]
}

In this example, the image belongs to the iNatAg_subset dataset. To use the provided annotations, users must:

download the original source datasets listed in Appendix A of the paper
create a local datasets_sorted/ directory
place each source dataset under the matching dataset-name subfolder used in the JSONL paths

Example expected layout:

text

datasets_sorted/
├── iNatAg_subset/
├── classification/
├── detection/
└── ...

Quickstart

python

import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration

BASE_MODEL_ID = "llava-hf/llava-onevision-qwen2-7b-ov-hf"
AGRICHAT_REPO = "boudiafA/AgriChat"

processor = AutoProcessor.from_pretrained(BASE_MODEL_ID)
base_model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    low_cpu_mem_usage=True,
)
model = PeftModel.from_pretrained(
    base_model,
    AGRICHAT_REPO,
)
model.eval()

image = Image.open("path/to/image.jpg").convert("RGB")
prompt = "What is shown in this agricultural image?"

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
device = next(model.parameters()).device
inputs = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=512, do_sample=False)

input_len = inputs["input_ids"].shape[1]
response = processor.tokenizer.decode(output_ids[0][input_len:], skip_special_tokens=True)
print(response.strip())

Performance Snapshot

AgriChat outperforms strong open-source generalist baselines on multiple agriculture benchmarks.

Benchmark	AgriChat
AgriMM	66.70 METEOR / 77.43 LLM Judge
PlantVillageVQA	19.52 METEOR / 74.26 LLM Judge
CDDM	39.59 METEOR / 69.94 LLM Judge
AGMMU	63.87 accuracy

Limitations

Performance depends on image quality and coverage of the training data.
The model can still make confident but incorrect statements.
Outputs should be reviewed carefully before use in real agricultural decision workflows.
The provided dataset/ annotations require the user to obtain the original source images separately.

Citation

bibtex

@article{boudiaf2026agrichat,
  title     = {AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding},
  author    = {Boudiaf, Abderrahmene and Hussain, Irfan and Javed, Sajid},
  journal   = {Submitted to Computers and Electronics in Agriculture},
  year      = {2026}
}