CropGuard - Crop Disease Classifier (38 classes)
ResNet50 fine-tuned on PlantVillage, exported to ONNX and dynamically quantised to INT8 for
CPU-only serving. Part of
CropGuard, an
end-to-end MLOps pipeline.
Results (held-out test set, n=8,125)
| Model | Accuracy | Macro-F1 | Note |
|---|
| fp32 | 0.9911 | 0.9865 | reference |
| INT8 (dynamic) | not evaluated | not evaluated | not served - see below |
Macro-F1 is the metric to read here, not accuracy: the dataset is imbalanced ~36x, so accuracy
is dominated by the largest classes.
cropguard.onnx (fp32) is the model that serves traffic. cropguard.static-int8.onnx is
published alongside it for comparison; the dynamic INT8 build is deliberately not
published.
Dynamic quantization was the wrong tool here. quantize_dynamic rewrites every Conv into
ConvInteger, which ONNX Runtime's CPU backend has no optimized kernel for: measured on an
Intel Alder Lake CPU it ran at 1567 ms/image against 19 ms for fp32, a 75x regression in
exchange for 4x less disk. It suits MatMul-dominated models (Transformers, RNNs), not CNNs.
Static quantization with a calibration set emits the optimized QLinearConv instead, and is
78 ms/image - 20x faster than dynamic and accuracy-neutral (0.9897 vs 0.9893 on a 3,000
image subset, 9 disagreements). It is still 3.2x slower than fp32 on the machine it was
measured on, and the reason is hardware rather than the graph: that CPU has no VNNI
instructions, without which ONNX Runtime emulates each INT8 multiply-accumulate in several
AVX2 instructions. Server CPUs usually do have VNNI, where the ranking may reverse - which is
exactly why both files are here to be benchmarked on whatever hardware you are running.
The split is grouped by leaf, and that matters
PlantVillage contains 54,305 images of only ~7,600 distinct physical leaves -
roughly 7 photographs of each. A standard per-image stratified split scatters those
near-duplicates across train and test, so a model can score well by memorising leaf identity
rather than learning disease morphology. Measured on this dataset, a naive split leaves
74.2% of test images sharing a leaf with training.
This model was trained on a leaf-grouped split instead:
| Naive stratified | Grouped (used here) |
|---|
| test images sharing a leaf with train | 74.2% | 0.0% |
| train / val / test | - | 38,008 / 8,172 / 8,125 |
So the accuracy above is measured on a holdout with no leaf overlap. Note that it is not
much lower than typically published PlantVillage figures - the honest reading is that this
dataset is genuinely easy, not that leakage was inflating everything.
Intended use
Identifying disease on single leaves photographed against a plain background, matching the
PlantVillage capture protocol.
Limitations - read before deploying this
- Lab images, not field images. Every training image is a detached leaf on a uniform
background under controlled lighting. Real photographs from a farm - variable lighting,
occlusion, multiple leaves, soil backgrounds - are a different distribution, and published
work on this dataset reports large drops there. This model has not been evaluated on
field photographs.
- Per-class metrics for rare classes are noisy. Eight classes have fewer than 100 test
images; the smallest (
Potato___healthy) has 24. A recall of 0.833 there is 4 mistakes, and
its confidence interval spans roughly +/-15 points. Do not read those per-class numbers as
precise.
- The raw softmax is not calibrated. Training used label smoothing (0.1), which caps
achievable confidence at (1-eps) + eps/K = 0.9026 and leaves the model systematically
under-confident. Temperature scaling fitted on the validation split (T = 0.591) cuts
expected calibration error from 0.0895 to 0.0036 and is applied in the serving path, but the
ONNX graph published here emits logits - apply the temperature yourself, or the
probabilities you compute from it will understate.
uncertainty is predictive entropy, not epistemic uncertainty. It cannot distinguish an
ambiguous input from one far outside the training distribution - an out-of-distribution
image can produce confidently wrong output with low entropy.- 38 classes across 14 crops only. Anything outside that set is silently forced into one of
them.
Training
ResNet50 (timm, ImageNet-pretrained), 224x224, batch 64, AdamW (lr 3e-4, weight decay 1e-4),
cosine schedule, label smoothing 0.1, medium augmentation, 12 epochs, mixed precision.
Checkpoint selected on val_f1_macro.
Usage
1import numpy as np, onnxruntime as ort
2from huggingface_hub import hf_hub_download
3
4path = hf_hub_download("XiElonMAsk/cropguard-models", "cropguard.onnx")
5session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
6
7# Preprocessing must match training: resize short side to 256, centre-crop 224,
8# scale to [0,1], normalise with ImageNet mean/std, NCHW.
9logits = session.run(["logits"], {"input": batch})[0]
cropguard.serving.model_loader in the repo implements exactly that preprocessing.
Citation
Dataset: Mohanty, Hughes & Salathe (2016), Using deep learning for image-based plant disease
detection, Frontiers in Plant Science.
