You can deploy this model and start earning money today!

Basavaraj000/
pancake

Name: pancake
Availability: InStock

openai/clip-vit-base-patch32

finetune

mit

Views

No views yet

Multimodal CLIP Classifier

A multimodal classification model that takes an image + text pair and predicts one of N classes. Built by fine-tuning a small classification head on top of frozen CLIP image and text embeddings.

⚠️ Placeholder values below (repo id, class names, dataset, metrics) — fill these in with your actual details before publishing.

Model Details

Base model: openai/clip-vit-base-patch32
Architecture: CLIP image + text encoders (frozen) → concatenated embeddings → 2-layer MLP classification head
Task: Multimodal classification ({{NUM_CLASSES}} classes)
Input: one image + one short text string
Output: class label + per-class probabilities
Repo: {{your-username}}/{{your-model-name}}

Intended Use

Describe here what the image+text pairs represent in your use case, and what the classes mean (e.g. "classifies product photos + descriptions into category X/Y/Z").

Training Data

Dataset: {{dataset name / link}}
Size: {{N examples}}
Split: {{train/val/test sizes}}

Training Procedure

Backbone frozen, only the classification head trained
Optimizer: AdamW, lr=1e-4
Loss: Cross-entropy
Epochs: {{N}}
Batch size: {{N}}

Evaluation Results

Metric	Value
Accuracy	{{TBD}}
F1 (macro)	{{TBD}}

How to Use

python

from model import MultimodalClassifier
from inference import load_model, predict

model = load_model("checkpoints/model.pt", num_classes=4,
                    base_model_name="openai/clip-vit-base-patch32")

result = predict(model, image_path="example.jpg", text="a description")
print(result["predicted_class"], result["confidence"])

Limitations

Inherits any biases present in the pretrained CLIP model.
Frozen backbone means performance is capped by CLIP's original embedding quality — unfreeze (freeze_backbone=False) and fine-tune end-to-end for more task-specific performance if you have enough data.

Files

model.py — model definition (MultimodalClassifier)
train.py — fine-tuning script
inference.py — single-example inference / CLI
requirements.txt — dependencies

Basavaraj000/pancake