Marqo-FashionSigLIP leverages Generalised Contrastive Learning (
GCL) which allows the model to be trained on not just text descriptions but also categories, style, colors, materials, keywords and fine-details to provide highly relevant search results on fashion products.
The model was fine-tuned from ViT-B-16-SigLIP (webli).
The model can be seamlessly used with
OpenCLIP by
1import open_clip
2model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionSigLIP')
3tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionSigLIP')
4
5import torch
6from PIL import Image
7
8image = preprocess_val(Image.open("docs/fashion-hippo.png")).unsqueeze(0)
9text = tokenizer(["a hat", "a t-shirt", "shoes"])
10
11with torch.no_grad(), torch.cuda.amp.autocast():
12 image_features = model.encode_image(image)
13 text_features = model.encode_text(text)
14 image_features /= image_features.norm(dim=-1, keepdim=True)
15 text_features /= text_features.norm(dim=-1, keepdim=True)
16
17 text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
18
19print("Label probs:", text_probs)
Average evaluation results on 6 public multimodal fashion datasets (
Atlas,
DeepFashion (In-shop),
DeepFashion (Multimodal),
Fashion200k,
KAGL, and
Polyvore) are reported below:
