This repository contains one LoRA adapter for Qwen/Qwen3-VL-4B-Instruct in PEFT and llama.cpp GGUF formats. Version 2 prioritizes Food-101 dish recognition and adds experimental visible-ingredient and rough-gram output.

Nutrition totals are not predicted directly by this adapter. Beforeat uses the recognized ingredients and estimated portions as inputs to a separate USDA FoodData Central lookup and rules layer for calories and macronutrients.

Current Files

adapter_model.safetensors: v2 PEFT weights
adapter_config.json: rank 8, alpha 16 PEFT configuration
beforeat-food-nutrition-vision-lora-v2-f16.gguf: v2 F16 llama.cpp adapter
blend_metadata.json: reproducible v2 weight-interpolation metadata
beforeat-food-nutrition-vision-lora-f16.gguf: historical v1 GGUF release

Only one adapter is loaded at runtime. The v1 file remains available for rollback; new integrations should use the versioned v2 file.

Prompts

Dish recognition:

Identify the dish in this image. Reply with only the exact Food-101 class name.

Experimental visible ingredients and portions:

Estimate the visible ingredients and their portions in grams. Reply only with a JSON array of objects using keys "name" and "grams".

Evaluation

Task	Evaluation	Result
Food-101 dish recognition	500-image held-out set, strict	438/500 (87.60%)
Food-101 dish recognition	Alias/near matching	440/500 (88.00%)
Visible ingredients	Nutrition5k, 100 images	44.51% mean recall
Matched ingredient portions	Nutrition5k, 100 images	52.93% mean recall
Matched ingredient grams	Nutrition5k, 85 parsed rows	43.34 g MAE

The previous food-only release scored 434/500 (86.80%) strict and 438/500 (87.60%) with aliases on the same Food-101 evaluation.

The ingredient/portion results are experimental and should not be interpreted as direct nutrition accuracy. Hidden ingredients, cooking oils, recipe details, and true serving mass cannot be reliably recovered from one image.

Adapter Construction

The v2 weights are a linear interpolation between the 2,500-step Food-101 LoRA and the continued 4,000-step ingredient/portion LoRA. The nutrition checkpoint weight is 0.25. Both inputs use the same rank, alpha, tensor names, and target modules, producing a single 504-tensor LoRA adapter.

Rank: 8
Alpha: 16
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

llama.cpp GGUF Adapter

Base model: Qwen/Qwen3-VL-4B-Instruct
Required base GGUF: Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf
Base GGUF source: unsloth/Qwen3-VL-4B-Instruct-GGUF
Required projector: mmproj-F16.gguf from the same base GGUF repository
Recommended adapter scale: 1.0
llama.cpp commit: 00fa7cb284cbf133fc426733bd64238a3588a33e
GGUF tensor count: 504
GGUF file size: 33065152 bytes
SHA-256: ac31fb420d91fd8a4149e694c4da540db6208889c53f86b383069a6422a01f62

Conversion command:

bash

PYTHONPATH=tools/llama.cpp \
python tools/llama.cpp/convert_lora_to_gguf.py \
  adapters/qwen3vl4b_food_nutrition_blend_w025 \
  --base models/Qwen3-VL-4B-Instruct-bf16-remapped \
  --outfile adapters/qwen3vl4b_food_nutrition_blend_w025/beforeat-food-nutrition-vision-lora-v2-f16.gguf \
  --outtype f16 \
  --verbose

Runtime command:

bash

llama-mtmd-cli \
  -m Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj mmproj-F16.gguf \
  --image food.jpg \
  --lora-scaled beforeat-food-nutrition-vision-lora-v2-f16.gguf:1.0 \
  -p "Identify the dish in this image. Reply with only the exact Food-101 class name." \
  --jinja --temp 0 -n 16 -c 4096 --image-min-tokens 1024

Validation used the same command with --device none -ngl 0 --no-mmproj-offload --no-op-offload --no-kv-offload --fit off after the local Metal command queue was unavailable. llama.cpp accepted all adapter tensors with no missing, incompatible, or unexpected tensor errors. The Unsloth UD-Q4_K_XL base and its existing mmproj-F16.gguf are compatible and reusable.

Runtime Spot Checks

Expected	PyTorch/PEFT v2	Base GGUF	GGUF + v2 LoRA
prime rib	prime rib	roast beef	prime rib
bread pudding	panna cotta	dessert	panna cotta
chocolate cake	chocolate mousse	Chocolate Cake	chocolate mousse

The matching PEFT and GGUF+LoRA predictions demonstrate conversion fidelity, including the same errors.

Limitations and Use

Food recognition is specialized around the 101 Food-101 classes and may not generalize to arbitrary dishes, brands, mixed plates, or regional variants.
Ingredient names and grams are rough visual estimates, not measured portions.
Use a separate food-composition database and calculation layer for nutrition.
Food-101 and Nutrition5k were used for research and prototyping; review their terms before commercial training or redistribution workflows.
This adapter is not medical or dietary advice.

fatsam13/beforeat-food-nutrition-vision-lora

Beforeat Food Nutrition Vision LoRA