Base: unsloth/gemma-4-12b-it
HuggingFace: yusasif/nutrimama-vision (branch v2-12b-q8)
Task: Given a food image, name the exact Nigerian dish(es) visible.

The model is finetuned with LoRA (r=32) on vision and language layers using Unsloth for memory-efficient training. The final checkpoint is merged to float16 and optionally quantized to 8-bit for serving.

Dataset

~600 image-text pairs spanning 165+ Nigerian food classes — including:

Swallows: Eba, Iyan (pounded yam), Fufu, Amala, Tuwon Shinkafa, Tuwon Masara, Nni Ede
Soups & stews: Egusi, Ogbono, Efo Riro, Gbegiri, Ewedu, Afang, Edikang Ikong, Banga, Ofe Nsala, Okra, Bitterleaf, Ofe Oha, Miyan Taushe, and more
Rice dishes: Jollof rice, Ofada rice, Fried rice, Enugu Jollof rice
Beans: Ewa Aganyin, Akara, Moi Moi, Okpa, Kosai
Snacks & street food: Suya, Kilishi, Dodo, Dundun, Boli, Puff Puff, Kuli Kuli
Northern dishes: Dambun Shinkafa, Funbau, Miyan Geda, Kunun Aya, Kunun Tsamiya
Igbo specialties: Abacha, Nkwobi, Ofe Owerri, Ugba Ukpaka, Ukwa, Achicha Ede
Fruits & drinks: Agbalumo, Nono, Kunun Gyada

Images were collected from multiple dataset batches (nutrimama_dataset, nutrimama_dataset-1, nutrimama_dataset-2, nutrimama_dataset-3) and enriched with detailed Gemma-generated captions (finetune_dataset_gemma.json).

Training

Setting	Value
Base model	`unsloth/gemma-4-12b-it`
LoRA rank	32
LoRA alpha	32
Target modules	all-linear (vision + language layers)
Epochs	5
Learning rate	1e-5
LR scheduler	cosine
Batch size	1 × 8 grad accum = effective 8
Optimizer	adamw_8bit
Precision	bfloat16
Max sequence length	2048

Training was run on a single GPU using Unsloth's 4-bit quantized loading with gradient checkpointing.

Inference

python

from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL_PATH = "yusasif/nutrimama-vision"  # or local path to merged model

processor = AutoProcessor.from_pretrained(MODEL_PATH)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_PATH,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

image = Image.open("your_food_image.jpg").convert("RGB")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Identify the exact Nigerian food in this image by name."},
            {"type": "image", "image": image},
        ],
    }
]

input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=input_text, images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=1.0,
        top_p=0.95,
        top_k=64,
    )

answer = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)

Serving (vLLM)

python gemma-vision.py # starts an OpenAI-compatible API on port 8002

The server loads the v2-12b-q8 branch and exposes the model as gemma-4-finetuned.

Files

File	Description
`nutrimam.py`	Training script (12B model)
`nutrimam_v2.py`	Training script (4B model)
`gemma-vision.py`	vLLM inference server
`test_nutrimama.py`	Local inference test
`dataset_download_registry.json`	Primary image-caption dataset (~398 records)
`finetune_dataset_gemma.json`	Gemma-enriched captions (~202 records)
`efficientnet_labels.json`	Per-image food class labels

Related Projects

yusasif/Nigerian-language-tts — multilingual TTS for Nigerian languages
yusasif/nutrimama-vision-v2 — 4B model variant

yusasif/Nigerian-food-recognition-v2

nutrimama-vision

Model