You can deploy this model and start earning money today!

sergiudanstan/
smolvlm2-500m-foodextract-lora

Name: smolvlm2-500m-foodextract-lora
Availability: InStock

mrdbourke/FoodExtract-1k-Vision

HuggingFaceTB/SmolVLM2-500M-Video-Instruct

adapter

apache-2.0

Views

No views yet

SmolVLM2-500M Food Extractor (LoRA)

LoRA fine-tune of HuggingFaceTB/SmolVLM2-500M-Video-Instruct on mrdbourke/FoodExtract-1k-Vision.

Task: given a food image, extract structured JSON with the visible food and drink items.

Output format

json

{
  "is_food": 1,
  "image_title": "salad",
  "food_items": ["tomato", "lettuce", "cheese"],
  "drink_items": []
}

Usage

python

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

base = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
processor = AutoProcessor.from_pretrained(base)
processor.image_processor.do_image_splitting = False
model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.float32)
model = PeftModel.from_pretrained(model, "sergiudanstan/smolvlm2-500m-foodextract-lora").merge_and_unload()

prompt = ("Extract all food and drink items from this image. Respond ONLY with "
          "JSON using the keys: is_food (0 or 1), image_title (string), "
          "food_items (list of strings), drink_items (list of strings).")
messages = [{"role": "user", "content": [
    {"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[[your_pil_image]], return_tensors="pt")
print(processor.batch_decode(model.generate(**inputs, max_new_tokens=256)))

Fine-tuned with PEFT LoRA (r=16, alpha=32) on the SmolVLM2 text decoder projections.