Marqo-FashionCLIP and Marqo-FashionSigLIP outperform the previous state-of-the-art fashion CLIP models (see results below).
Marqo-FashionCLIP leverages Generalised Contrastive Learning (
GCL) which allows the model to be trained on not just text descriptions but also categories, style, colors, materials, keywords and fine-details to provide highly relevant search results on fashion products.
The model was fine-tuned from ViT-B-16 (laion2b_s34b_b88k).
1from transformers import AutoModel, AutoProcessor
2model = AutoModel.from_pretrained('Marqo/marqo-fashionCLIP', trust_remote_code=True)
3processor = AutoProcessor.from_pretrained('Marqo/marqo-fashionCLIP', trust_remote_code=True)
4
5import torch
6from PIL import Image
7
8image = [Image.open("docs/fashion-hippo.png")]
9text = ["a hat", "a t-shirt", "shoes"]
10processed = processor(text=text, images=image, padding='max_length', return_tensors="pt")
11
12with torch.no_grad():
13 image_features = model.get_image_features(processed['pixel_values'], normalize=True)
14 text_features = model.get_text_features(processed['input_ids'], normalize=True)
15
16 text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
17
18print("Label probs:", text_probs)
19# [0.99990773, 0.00006382, 0.00002847]
The model can be seamlessly used with
OpenCLIP by
1import open_clip
2model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionCLIP')
3tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionCLIP')
4
5import torch
6from PIL import Image
7
8image = preprocess_val(Image.open("docs/fashion-hippo.png")).unsqueeze(0)
9text = tokenizer(["a hat", "a t-shirt", "shoes"])
10
11with torch.no_grad(), torch.cuda.amp.autocast():
12 image_features = model.encode_image(image, normalize=True)
13 text_features = model.encode_text(text, normalize=True)
14
15 text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
16
17print("Label probs:", text_probs)
18# [0.9998498302475922, 0.000119267522939106, 0.000030902229468640687]
You can also run the model in JavaScript with the
Transformers.js library.
1import { CLIPTextModelWithProjection, CLIPVisionModelWithProjection, AutoTokenizer, AutoProcessor, RawImage, softmax, dot } from '@huggingface/transformers';
2
3const model_id = 'Marqo/marqo-fashionCLIP';
4
5// Load tokenizer and text model
6const tokenizer = await AutoTokenizer.from_pretrained(model_id);
7const text_model = await CLIPTextModelWithProjection.from_pretrained(model_id);
8
9// Load processor and vision model
10const processor = await AutoProcessor.from_pretrained(model_id);
11const vision_model = await CLIPVisionModelWithProjection.from_pretrained(model_id);
12
13// Run tokenization
14const texts = ['a hat', 'a t-shirt', 'shoes'];
15const text_inputs = tokenizer(texts, { padding: 'max_length', truncation: true });
16
17// Compute text embeddings
18const { text_embeds } = await text_model(text_inputs);
19
20// Read image and run processor
21const image = await RawImage.read('https://raw.githubusercontent.com/marqo-ai/marqo-FashionCLIP/main/docs/fashion-hippo.png');
22const image_inputs = await processor(image);
23
24// Compute vision embeddings
25const { image_embeds } = await vision_model(image_inputs);
26
27// Compute similarity scores
28const normalized_text_embeds = text_embeds.normalize().tolist();
29const normalized_image_embeds = image_embeds.normalize().tolist()[0];
30
31const text_probs = softmax(normalized_text_embeds.map((text_embed) =>
32 100.0 * dot(normalized_image_embeds, text_embed)
33));
34console.log(text_probs);
35// [0.9998498302475922, 0.000119267522939106, 0.000030902229468640687]
Average evaluation results on 6 public multimodal fashion datasets (
Atlas,
DeepFashion (In-shop),
DeepFashion (Multimodal),
Fashion200k,
KAGL, and
Polyvore) are reported below:
