Views
No views yet
[!IMPORTANT] This is a locally patched copy of nomic-ai/nomic-embed-vision-v1.5. See Local Modifications below.
transformers 4.x and fails to load under transformers 5.x (tested with 5.13.1). The following changes were made on 2026-07-14 to make it work without downgrading/pinning transformers. We deliberately patch the model in-repo rather than pin dependencies, since no further upstream updates to the remote code are expected.auto_map in config.json points to code hosted on the nomic-ai/nomic-bert-2048 Hub repo, which transformers downloads at load time. That code is incompatible with transformers 5.x, so configuration_hf_nomic_bert.py and modeling_hf_nomic_bert.py were copied into this directory and auto_map now references the local files:1"auto_map": {
2 "AutoConfig": "configuration_hf_nomic_bert.NomicBertConfig",
3 "AutoModel": "modeling_hf_nomic_bert.NomicVisionModel"
4}nomic-bert-2048 code are no longer picked up automatically.post_init() call in NomicVisionModel (transformers 5.x compatibility)transformers 5.x requires PreTrainedModel.post_init() to run during model construction — among other things it sets all_tied_weights_keys, which the weight-loading machinery accesses. The upstream NomicVisionModel.__init__ never calls it (the text-model classes in the same file do), causing:AttributeError: 'NomicVisionModel' object has no attribute 'all_tied_weights_keys'self.post_init() added at the end of NomicVisionModel.__init__ in modeling_hf_nomic_bert.py.transformers 5.x constructs models on the meta device, so any buffer values computed in a module's __init__ are discarded. The nomic code computes several non-persistent buffers there (norm_factor in the attention modules, the rotary pos_embed/bands in NomicVisionRotaryEmbeddingCat, inv_freq/scale in NomicBertRotaryEmbedding). These are not stored in the checkpoint, so after loading they contained uninitialized garbage — norm_factor was 0.0, making attention divide by zero and producing all-NaN embeddings (the model loaded without any error).modeling_hf_nomic_bert.py:_recompute_nonpersistent_buffers(module) that recomputes these buffers with the same formulas the __init__ methods use._init_weights on both NomicBertPreTrainedModel and NomicVisionPreTrainedModel — the transformers 5 loading machinery calls _init_weights for modules whose tensors were not found in the checkpoint. Both classes need the override because NomicBertBlock subclasses NomicBertPreTrainedModel, and transformers dispatches initialization inside each PreTrainedModel subtree to that class's own _init_weights (an override only on the vision top-level class never reaches the attention layers inside the blocks).NomicVisionRotaryEmbeddingCat.__init__ now stores self.linear_bands, needed for the recomputation.onnx/model.onnx) with cosine similarity 1.0 (max element diff ~2e-7).n_inner type fix in config.json"n_inner": 2048.0 (a float). Newer transformers/huggingface_hub strictly validate config field types and require int or None, raising:StrictDataclassFieldValidationError: Validation error for field 'n_inner'"n_inner": 2048.model.safetensors is stored via git-lfs; after cloning, run git lfs pull in this directory to fetch the actual weights (a 134-byte pointer file will otherwise cause SafetensorError: header too large).einops package at runtime.nomic-embed-vision-v1.5 is a high performing vision embedding model that shares the same embedding space as nomic-embed-text-v1.5.| Name | Imagenet 0-shot | Datacomp (Avg. 38) | MTEB |
|---|---|---|---|
nomic-embed-vision-v1.5 | 71.0 | 56.8 | 62.28 |
nomic-embed-vision-v1 | 70.7 | 56.7 | 62.39 |
| OpenAI CLIP ViT B/16 | 68.3 | 56.3 | 43.82 |
| Jina CLIP v1 | 59.1 | 52.2 | 60.1 |
nomic Python client is as easy as1from nomic import embed
2import numpy as np
3
4output = embed.image(
5 images=[
6 "image_path_1.jpeg",
7 "image_path_2.png",
8 ],
9 model='nomic-embed-vision-v1.5',
10)
11
12print(output['usage'])
13embeddings = np.array(output['embeddings'])
14print(embeddings.shape)
contrastors repositorynomic-embed-text requires prefixes and so, when using Nomic Embed in multimodal RAG scenarios (e.g. text to image retrieval),
you should use the search_query: prefix.1import torch
2import torch.nn.functional as F
3from transformers import AutoTokenizer, AutoModel, AutoImageProcessor
4from PIL import Image
5import requests
6
7processor = AutoImageProcessor.from_pretrained("xhresko/nomic-embed-vision-v1.5-patched")
8vision_model = AutoModel.from_pretrained("xhresko/nomic-embed-vision-v1.5-patched", trust_remote_code=True)
9
10url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
11image = Image.open(requests.get(url, stream=True).raw)
12
13inputs = processor(image, return_tensors="pt")
14
15img_emb = vision_model(**inputs).last_hidden_state
16img_embeddings = F.normalize(img_emb[:, 0], p=2, dim=1)1
2def mean_pooling(model_output, attention_mask):
3 token_embeddings = model_output[0]
4 input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
5 return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
6
7sentences = ['search_query: What are cute animals to cuddle with?', 'search_query: What do cats look like?']
8
9tokenizer = AutoTokenizer.from_pretrained('nomic-ai/nomic-embed-text-v1.5')
10text_model = AutoModel.from_pretrained('nomic-ai/nomic-embed-text-v1.5', trust_remote_code=True)
11text_model.eval()
12
13encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
14
15with torch.no_grad():
16 model_output = text_model(**encoded_input)
17
18text_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
19text_embeddings = F.layer_norm(text_embeddings, normalized_shape=(text_embeddings.shape[1],))
20text_embeddings = F.normalize(text_embeddings, p=2, dim=1)
21
22print(torch.matmul(img_embeddings, text_embeddings.T))