ColQwen is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features.
It is a
Qwen2.5-VL-3B extension that generates
ColBERT- style multi-vector representations of text and images.
It was introduced in the paper
ColPali: Efficient Document Retrieval with Vision Language Models and first released in
this repositoryThis model takes dynamic image resolutions in input and does not resize them, changing their aspect ratio as in ColPali.
Maximal resolution is set so that 768 image patches are created at most. Experiments show clear improvements with larger amounts of image patches, at the cost of memory requirements.
We train models use low-rank adapters (
LoRA)
with
alpha=128 and
r=128 on the transformer layers from the language model,
as well as the final randomly initialized projection layer, and use a
paged_adamw_8bit optimizer.
We train on an 8xA100 GPU setup with distributed data parallelism (via accelerate), a learning rate of 2e-4 with linear decay with 1% warmup steps, batch size per device is 32, gradient accumulation steps are 2, in
bfloat16 format
The code of Qwen2.5-VL has been in the latest Hugging face transformers and we advise you to build from source with command:
1import torch
2from PIL import Image
3
4from colpali_engine.models import ColQwen2_5, ColQwen2_5_Processor
5
6model = ColQwen2_5.from_pretrained(
7 "yydxlv/colqwen2.5-7b-v0.1",
8 torch_dtype=torch.bfloat16,
9 device_map="cuda:0", # or "mps" if on Apple Silicon
10 ).eval()
11processor = ColQwen2_5_Processor.from_pretrained("yydxlv/colqwen2.5-7b-v0.1")
12
13# Your inputs
14images = [
15 Image.new("RGB", (32, 32), color="white"),
16 Image.new("RGB", (16, 16), color="black"),
17]
18queries = [
19 "Is attention really all you need?",
20 "What is the amount of bananas farmed in Salvador?",
21]
22
23# Process the inputs
24batch_images = processor.process_images(images).to(model.device)
25batch_queries = processor.process_queries(queries).to(model.device)
26
27# Forward pass
28with torch.no_grad():
29 image_embeddings = model(**batch_images)
30 query_embeddings = model(**batch_queries)
31
32scores = processor.score_multi_vector(query_embeddings, image_embeddings)
If you use this models from this organization in your research, please cite the original paper as follows:
1@misc{faysse2024colpaliefficientdocumentretrieval,
2 title={ColPali: Efficient Document Retrieval with Vision Language Models},
3 author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
4 year={2024},
5 eprint={2407.01449},
6 archivePrefix={arXiv},
7 primaryClass={cs.IR},
8 url={https://arxiv.org/abs/2407.01449},
9}
