UForm
Multi-Modal Inference Library
For Semantic Search Applications
UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
This is model card of the English only model with:
- 4 layers BERT (2 layers for unimodal encoding and rest layers for multimodal encoding)
- ViT-S/16 (image resolution is 224x224)
If you need Multilingual model, check
this.
Evaluation
The following metrics were obtained with multimodal re-ranking (text-to-image retrieval):
| Dataset | Recall@1 | Recall@5 | Recall@10 |
|---|
| Zero-Shot Flickr | 0.565 | 0.790 | 0.860 |
| Zero-Shot MS-COCO | 0.281 | 0.525 | 0.645 |
ImageNet-Top1: 0.361
ImageNet-Top5: 0.608
Installation
Usage
To load the model:
1import uform
2
3model, processor = uform.get_model_onnx('unum-cloud/uform-vl-english-small', device='cpu', dtype='fp32')
To encode data:
1from PIL import Image
2
3text = 'a small red panda in a zoo'
4image = Image.open('red_panda.jpg')
5
6image_data = processor.preprocess_image(image)
7text_data = processor.preprocess_text(text)
8
9image_features, image_embedding = model.encode_image(image_data, return_features=True)
10text_features, text_embedding = model.encode_text(text_data, return_features=True)
11score, joint_embedding = model.encode_multimodal(
12 image_features=image_features,
13 text_features=text_features,
14 attention_mask=text_data['attention_mask'],
15 return_scores=True
16)
There are two options to calculate semantic compatibility between an image and a text: cosine similarity and
Matching Score.
Cosine Similarity
The similarity will belong to the [-1, 1] range, 1 meaning the absolute match.
Pros:
- Computationally cheap.
- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
- Suitable for retrieval in large collections.
Cons:
- Takes into account only coarse-grained features.
Matching Score
Unlike cosine similarity, unimodal embedding are not enough.
Joint embedding will be needed and the resulting score will belong to the [0, 1] range, 1 meaning the absolute match.
Pros:
- Joint embedding captures fine-grained features.
- Suitable for re-ranking – sorting retrieval result.
Cons:
- Resource-intensive.
- Not suitable for retrieval in large collections.
