Views
No views yet
Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.
colpali-engine:pip install colpali-engine>=0.3.0,<0.4.01from typing import cast
2
3import torch
4from PIL import Image
5
6from colpali_engine.models import ColPali, ColPaliProcessor
7
8model_name = "ahmed-masry/ColMate-3B"
9
10model = ColPali.from_pretrained(
11 model_name,
12 torch_dtype=torch.bfloat16,
13 device_map="cuda:0", # or "mps" if on Apple Silicon
14).eval()
15
16processor = ColPaliProcessor.from_pretrained(model_name)
17
18# Your inputs
19images = [
20 Image.new("RGB", (32, 32), color="white"),
21 Image.new("RGB", (16, 16), color="black"),
22]
23queries = [
24 "Is attention really all you need?",
25 "Are Benjamin, Antoine, Merve, and Jo best friends?",
26]
27
28# Process the inputs
29batch_images = processor.process_images(images).to(model.device)
30batch_queries = processor.process_queries(queries).to(model.device)
31
32# Forward pass
33with torch.no_grad():
34 image_embeddings = model(**batch_images)
35 query_embeddings = model(**batch_queries)
36
37scores = processor.score_multi_vector(query_embeddings, image_embeddings)1@inproceedings{masry-etal-2025-colmate,
2 title = "{C}ol{M}ate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval",
3 author = "Masry, Ahmed and
4 Thakkar, Megh and
5 Bechard, Patrice and
6 Madhusudhan, Sathwik Tejaswi and
7 Awal, Rabiul and
8 Mishra, Shambhavi and
9 Suresh, Akshay Kalkunte and
10 Daruru, Srivatsava and
11 Hoque, Enamul and
12 Gella, Spandana and
13 Scholak, Torsten and
14 Rajeswar, Sai",
15 editor = "Potdar, Saloni and
16 Rojas-Barahona, Lina and
17 Montella, Sebastien",
18 booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
19 month = nov,
20 year = "2025",
21 address = "Suzhou (China)",
22 publisher = "Association for Computational Linguistics",
23 url = "https://aclanthology.org/2025.emnlp-industry.145/",
24 doi = "10.18653/v1/2025.emnlp-industry.145",
25 pages = "2071--2080",
26 ISBN = "979-8-89176-333-3",
27 abstract = "Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61{\%} improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks."
28}