A LoRA fine-tune of SmolVLM2-500M-Video-Instruct, merged into a standalone model. Given a photo of ingredients arranged on a surface, it returns a JSON list of the ingredients it recognizes. Built as an end-to-end ML portfolio project (data → fine-tune → inference endpoint → web app).

V2 change: the vocabulary was expanded from 51 to ~353 ingredient classes by unioning the original 51-class set with a 316-class dataset (14 ingredients shared and merged to a single canonical label). This directly targets V1's main failure mode — padding the output with frequent in-vocab guesses whenever an out-of-vocabulary ingredient appeared — by giving many of those previously-unknown items a real label.

Task: image → JSON ingredient list
Classes: ~353 single ingredients (51 legacy + 316 new − 14 overlapping, merged)
Adapter: LoRA on q/k/v/o + gate/up/down projections, merged
Trained from base, not continued from V1 (avoids forgetting the non-overlapping legacy classes)

Usage

python

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

repo = "LongGrainRice/kimchi-test"
processor = AutoProcessor.from_pretrained(repo)
# bf16 needs an Ampere+ GPU (e.g. L4). On a T4 or older card, use torch.float16.
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to("cuda")

# Use the exact instruction the model was trained with — it keys on this wording.
INSTRUCTION = ("You are a food recognition assistant. List every food ingredient in this image. "
               'Respond ONLY with a JSON array of lowercase strings, e.g. ["milk", "eggs", "tomato"].')

image = Image.open("slab.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": INSTRUCTION}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[prompt], images=[[image]], return_tensors="pt", padding=True).to(model.device)

ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

De-duplicate the output before using it: sorted(set(...)).

Training data

Synthetic composites built by unioning two single-item sources, both center-cropped with an oval mask and pasted onto slab backgrounds with varied scale, rotation, and shadow; the label is the set of pasted items.

liamboyd1/singular-food-items — 51 classes, image-rich (~1.3k/class), capped per class for variety.
Scuccorese/food-ingredients-dataset — 316 classes, ~21 images/class, with a 12-category / 28-subcategory hierarchy.

The 14 ingredients shared between the two sets are normalized to a single canonical name so they don't fragment into duplicate classes. Because the two sources differ ~60× in per-class image count, the compositor samples classes uniformly when building scenes rather than sampling images uniformly — otherwise the image-rich legacy classes would dominate and the new vocabulary would rarely appear. A small set of real photos is mixed in and upweighted.

Training

Base: SmolVLM2-500M-Video-Instruct, LoRA (q/k/v/o + gate/up/down), merged after training.
~3 epochs, effective batch 8, cosine schedule, gradient checkpointing.
bf16 on L4 (24 GB); the pipeline auto-falls back to fp16 + smaller batch on a T4.

Limitations

Vocabulary is fixed at ~353 classes. The expansion covers many items V1 couldn't name, but anything outside the set still has no label.
Pantry staples aren't detected. Salt, oil, sugar, flour and similar are generally not visible as distinct items and aren't in scope; the downstream recipe step introduces them separately, tagged as pantry/extra, so users aren't misled about what was actually detected.
New classes are data-thin (~21 images each). Visually adjacent ingredients (e.g. kale vs collard greens, scallion vs leek) can be confused or both emitted for one item. This is a different, milder error mode than V1's out-of-vocab padding, but it means precision on fine-grained near-neighbors is the weak spot. Near-synonyms may be worth collapsing to a coarser label for a given use case.
Always de-duplicate the output (sorted(set(...))).

Best used on scenes composed from the known classes. A base-vs-fine-tuned comparison script is included in the project repo for evaluating on your own slab photos.