DINOv3-food is a food image recognition model fine-tuned from facebook/dinov3-vitl16-pretrain-lvd1689m on TSOTSA-Img, a merged food image dataset built from AFD, FruitVeg-81, Food-101, and UECFood256.

The model predicts 389 food categories and uses a DINOv3 ViT-L/16 backbone with a lightweight linear classification head.

Dataset

TSOTSA-Img is the merged dataset used for food recognition in this work. It is split into training and test subsets:

Training split: used to fine-tune the model.
Test split: used for final evaluation.

The merged dataset combines food images and labels from:

AFD
FruitVeg-81
Food-101
UECFood256

Training

The selected checkpoint was fine-tuned for 8 epochs.

Setting	Value
Base model	`facebook/dinov3-vitl16-pretrain-lvd1689m`
Backbone	DINOv3 ViT-L/16
Number of labels	389
Epochs	8
Batch size	16
Learning rate	`2e-5`
Weight decay	`0.01`
Warmup ratio	`0.05`
Validation selection	best validation behavior, with emphasis on validation loss

Validation metrics for the selected run:

Metric	Value
Validation loss	0.1100
Accuracy	0.9731
Macro-F1	0.9727
Top-5 accuracy	0.9968

Evaluation

Final evaluation was performed on the individual source datasets and on the merged TSOTSA-Img test split.

Dataset	Accuracy
FruitVeg-81	0.9976
AFD	0.9997
Food-101	0.9551
UECFood256	0.8215
TSOTSA-Img test	0.9062

For the TSOTSA-Img test split:

Metric	Value
Accuracy	0.9062
Macro-F1	0.9072

Model format

This repository stores a custom backbone-plus-classifier model:

backbone/: DINOv3 backbone saved with transformers.
classifier.pt: linear classification head.
classifier_config.json: label mappings and classifier metadata.
preprocessor_config.json: image preprocessing configuration.

Because this model uses a custom wrapper around the DINOv3 backbone, loading it with AutoModelForImageClassification.from_pretrained(...) is not sufficient. Use the project loader or reconstruct the wrapper before inference.

Usage

Example with the project inference class:

python

from inference.food_classifier import FoodClassifier

model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"
classifier = FoodClassifier(model_dir)

prediction = classifier.predict("path/to/food_image.jpg")
print(prediction)

Manual loading:

python

import json
import torch
from transformers import AutoImageProcessor, AutoModel
from finetuning.train_classifier import BackboneImageClassifier

model_dir = "model_saved/finetuning/facebook-dinov3-vitl16-pretrain-lvd1689m/epochs_8"

with open(f"{model_dir}/classifier_config.json", "r", encoding="utf-8") as f:
    classifier_config = json.load(f)

id2label = {
    int(label_id): label
    for label_id, label in classifier_config["id2label"].items()
}
label2id = {
    label: int(label_id)
    for label, label_id in classifier_config["label2id"].items()
}

backbone = AutoModel.from_pretrained(f"{model_dir}/backbone")
model = BackboneImageClassifier(
    backbone=backbone,
    num_labels=int(classifier_config["num_labels"]),
    id2label=id2label,
    label2id=label2id,
)

classifier_state = torch.load(f"{model_dir}/classifier.pt", map_location="cpu")
model.classifier.load_state_dict(classifier_state)
model.eval()

processor = AutoImageProcessor.from_pretrained(model_dir)

Intended use

This model is intended for food image recognition over the TSOTSA-Img label space. It can be used for research experiments, dataset benchmarking, and food recognition pipelines where the target labels overlap with the 389 supported categories.

Limitations

The model is restricted to the 389 labels in classifier_config.json.
Performance may degrade on food categories outside the TSOTSA-Img label space.
Predictions may be sensitive to ambiguous images, mixed dishes, heavy occlusion, or visually similar food categories.
The model card reports accuracy on the available benchmark splits and should not be interpreted as performance on all possible food domains.

anonymous-eval/food-recognition

DINOv3-food