ColNetraEmbed is a multilingual multimodal embedding model that encodes documents as multi-vector representations using the ColPali architecture. Each image patch is mapped to a contextualized embedding, enabling fine-grained matching between visual content and text queries through late interaction (MaxSim).
1import torch
2from PIL import Image
3from colpali_engine.models import ColGemma3, ColGemmaProcessor3
4
5# Load model and processor
6model_name = "Cognitive-Lab/ColNetraEmbed"
7model = ColGemma3.from_pretrained(
8 model_name,
9 torch_dtype=torch.bfloat16,
10 device_map="cuda",
11)
12processor = ColGemmaProcessor3.from_pretrained(model_name)
13
14# Load your images
15images = [
16 Image.open("document1.jpg"),
17 Image.open("document2.jpg"),
18]
19
20# Define queries
21queries = [
22 "What is the total revenue?",
23 "Show me the organizational chart",
24]
25
26# Process and encode
27batch_images = processor.process_images(images).to(model.device)
28batch_queries = processor.process_queries(queries).to(model.device)
29
30with torch.no_grad():
31 image_embeddings = model(**batch_images) # Shape: (num_images, num_patches, 128)
32 query_embeddings = model(**batch_queries) # Shape: (num_queries, num_tokens, 128)
33
34# Compute similarity scores using MaxSim
35scores = processor.score_multi_vector(
36 qs=query_embeddings,
37 ps=image_embeddings,
38) # Shape: (num_queries, num_images)
39
40# Get best matches
41for i, query in enumerate(queries):
42 best_idx = scores[i].argmax().item()
43 print(f"Query: '{query}' -> Best match: Image {best_idx + 1} (score: {scores[i, best_idx]:.2f})")
ColNetraEmbed achieves strong performance on multilingual document retrieval benchmarks. Evaluated on
Nayana-IR Bench (22 languages) and ViDoRe v2.
See our
paper for comprehensive evaluation and architectural comparisons.
1@misc{kolavi2025m3druniversalmultilingualmultimodal,
2 title={M3DR: Towards Universal Multilingual Multimodal Document Retrieval},
3 author={Adithya S Kolavi and Vyoman Jain},
4 year={2025},
5 eprint={2512.03514},
6 archivePrefix={arXiv},
7 primaryClass={cs.IR},
8 url={https://arxiv.org/abs/2512.03514}
9}
This model is released under the same license as the base Gemma3 model.
This work benefited from compute credits for training, inference, and evaluation provided by
Modal, acknowledged as a compute sponsor. Dataset curation and synthesis were supported by the
Meta LLaMA Impact Grant through our
Nayana initiative. We appreciate Meta for continued support of our research efforts at
CognitiveLab.
Built on top of the ColPali framework and Gemma3 architecture.
