ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features.
It is a
PaliGemma-3B extension that generates
ColBERT- style multi-vector representations of text and images.
It was introduced in the paper
ColPali: Efficient Document Retrieval with Vision Language Models and first released in
this repositoryThis model is built iteratively starting from an off-the-shelf
SigLIP model.
We finetuned it to create
BiSigLIP and fed the patch-embeddings output by SigLIP to an LLM,
PaliGemma-3B to create
BiPali.
One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to textual input (query).
This enables leveraging the
ColBERT strategy to compute interactions between text tokens and image patches, which enables a step-change improvement in performance compared to BiPali.
Our training dataset of 127,460 query-page pairs is comprised of train sets of openly available academic datasets (63%) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions (37%).
Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages. We explicitly verify no multi-page PDF document is used both
ViDoRe and in the train set to prevent evaluation contamination.
A validation set is created with 2% of the samples to tune hyperparameters.
All models are trained for 1 epoch on the train set. Unless specified otherwise, we train models in
bfloat16 format, use low-rank adapters (
LoRA)
with
alpha=32 and
r=32 on the transformer layers from the language model,
as well as the final randomly initialized projection layer, and use a
paged_adamw_8bit optimizer.
We train on an 8 GPU setup with data parallelism, a learning rate of 5e-5 with linear decay with 2.5% warmup steps, and a batch size of 32.
This version should not be used, it is solely the base version useful for deterministic LoRA initialization !
ColPali's vision language backbone model (PaliGemma) is under
gemma license as specified in its
model card. The adapters attached to the model are under MIT license.
If you use any datasets or models from this organization in your research, please cite the original dataset as follows:
1@misc{faysse2024colpaliefficientdocumentretrieval,
2 title={ColPali: Efficient Document Retrieval with Vision Language Models},
3 author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
4 year={2024},
5 eprint={2407.01449},
6 archivePrefix={arXiv},
7 primaryClass={cs.IR},
8 url={https://arxiv.org/abs/2407.01449},
9}
