JinaColBERT V2: A General-Purpose Multilingual Late Interaction Retriever.
JinaColBERT V2 (jina-colbert-v2) is a new model based on the JinaColBERT V1 that expands on the capabilities and performance of the jina-colbert-v1-en model. Like the previous release, it has Jina AI’s 8192 token input context and the improved efficiency, performance, and explainability of token-level embeddings and late interaction.
This new release adds new functionality and performance improvements:
Multilingual support for dozens of languages, with strong performance on major global languages.
Matryoshka embeddings, which allow users to trade between efficiency and precision flexibly.
Superior retrieval performance when compared to the English-only jina-colbert-v1-en.
jina-colbert-v2 is trained with flash attention, so einops is required and flash_attn is recommended. Without flash_attn the model falls back to PyTorch's native attention implementation. Beyond that, pick whichever inference library you prefer.
pip install -U einops flash_attn
Sentence Transformers
This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:
pip install "sentence-transformers>=6.0.0"
python
1from sentence_transformers import MultiVectorEncoder
23model = MultiVectorEncoder("jinaai/jina-colbert-v2", trust_remote_code=True)45query ="What does ColBERT do?"6documents =[7"ColBERT is a novel ranking model that adapts deep LMs for efficient retrieval.",8"Jina-ColBERT is a ColBERT-style model but based on JinaBERT so it can support both 8k context length, fast and accurate retrieval.",9]1011query_embeddings = model.encode_query(query)12document_embeddings = model.encode_document(documents)13print(query_embeddings.shape, document_embeddings[0].shape)14# (32, 128) (23, 128)1516# MaxSim late-interaction scoring (higher is more relevant)17scores = model.similarity(query_embeddings, document_embeddings)18print(scores)19# tensor([[23.2578, 21.5039]])
1from ragatouille import RAGPretrainedModel
23RAG = RAGPretrainedModel.from_pretrained("jinaai/jina-colbert-v2")4docs =[5"ColBERT is a novel ranking model that adapts deep LMs for efficient retrieval.",6"Jina-ColBERT is a ColBERT-style model but based on JinaBERT so it can support both 8k context length, fast and accurate retrieval.",7]8RAG.index(docs, index_name="demo")9query ="What does ColBERT do?"10results = RAG.search(query)
Stanford ColBERT
pip install colbert-ai
python
1from colbert.infra import ColBERTConfig
2from colbert.modeling.checkpoint import Checkpoint
34ckpt = Checkpoint("jinaai/jina-colbert-v2", colbert_config=ColBERTConfig())5docs =[6"ColBERT is a novel ranking model that adapts deep LMs for efficient retrieval.",7"Jina-ColBERT is a ColBERT-style model but based on JinaBERT so it can support both 8k context length, fast and accurate retrieval.",8]9query_vectors = ckpt.queryFromText(docs, bsize=2)
Evaluation Results
Retrieval Benchmarks
BEIR
NDCG@10
jina-colbert-v2
jina-colbert-v1
ColBERTv2.0
BM25
avg
0.531
0.502
0.496
0.440
nfcorpus
0.346
0.338
0.337
0.325
fiqa
0.408
0.368
0.354
0.236
trec-covid
0.834
0.750
0.726
0.656
arguana
0.366
0.494
0.465
0.315
quora
0.887
0.823
0.855
0.789
scidocs
0.186
0.169
0.154
0.158
scifact
0.678
0.701
0.689
0.665
webis-touche
0.274
0.270
0.260
0.367
dbpedia-entity
0.471
0.413
0.452
0.313
fever
0.805
0.795
0.785
0.753
climate-fever
0.239
0.196
0.176
0.213
hotpotqa
0.766
0.656
0.675
0.603
nq
0.640
0.549
0.524
0.329
MS MARCO Passage Retrieval
MRR@10
jina-colbert-v2
jina-colbert-v1
ColBERTv2.0
BM25
MSMARCO
0.396
0.390
0.397
0.187
Multilingual Benchmarks
MIRACLE
NDCG@10
jina-colbert-v2
mDPR (zero shot)
avg
0.627
0.427
ar
0.753
0.499
bn
0.750
0.443
de
0.504
0.490
es
0.538
0.478
en
0.570
0.394
fa
0.563
0.480
fi
0.740
0.472
fr
0.541
0.435
hi
0.600
0.383
id
0.547
0.272
ja
0.632
0.439
ko
0.671
0.419
ru
0.643
0.407
sw
0.499
0.299
te
0.742
0.356
th
0.772
0.358
yo
0.623
0.396
zh
0.523
0.512
mMARCO
MRR@10
jina-colbert-v2
BM-25
ColBERT-XM
avg
0.313
0.141
0.254
ar
0.272
0.111
0.195
de
0.331
0.136
0.270
nl
0.330
0.140
0.275
es
0.341
0.158
0.285
fr
0.335
0.155
0.269
hi
0.309
0.134
0.238
id
0.319
0.149
0.263
it
0.337
0.153
0.265
ja
0.276
0.141
0.241
pt
0.337
0.152
0.276
ru
0.298
0.124
0.251
vi
0.287
0.136
0.226
zh
0.302
0.116
0.246
Matryoshka Representation Benchmarks
BEIR
NDCG@10
dim=128
dim=96
dim=64
avg
0.599
0.591
0.589
nfcorpus
0.346
0.340
0.347
fiqa
0.408
0.404
0.404
trec-covid
0.834
0.808
0.805
hotpotqa
0.766
0.764
0.756
nq
0.640
0.640
0.635
MSMARCO
MRR@10
dim=128
dim=96
dim=64
msmarco
0.396
0.391
0.388
Other Models
Additionally, we provide the following embedding models, you can also use them for retrieval.
jina-clip-v1: English multimodal (text-image) embedding model.
Contact
Join our Discord community and chat with other community members about ideas.
@inproceedings{xiao-etal-2024-jina,
title = "{J}ina-{C}ol{BERT}-v2: A General-Purpose Multilingual Late Interaction Retriever",
author = {Jha, Rohan and
Wang, Bo and
G{\"u}nther, Michael and
Mastrapas, Georgios and
Sturua, Saba and
Mohr, Isabelle and
Koukounas, Andreas and
Wang, Mohammad Kalim and
Wang, Nan and
Xiao, Han},
editor = {S{\"a}lev{\"a}, Jonne and
Owodunni, Abraham},
booktitle = "Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.mrl-1.11/",
doi = "10.18653/v1/2024.mrl-1.11",
pages = "159--166",
abstract = "Multi-vector dense models, such as ColBERT, have proven highly effective in information retrieval. ColBERT`s late interaction scoring approximates the joint query-document attention seen in cross-encoders while maintaining inference efficiency closer to traditional dense retrieval models, thanks to its bi-encoder architecture and recent optimizations in indexing and search. In this paper, we introduce a novel architecture and a training framework to support long context window and multilingual retrieval. Leveraging Matryoshka Representation Loss, we further demonstrate that the reducing the embedding dimensionality from 128 to 64 has insignificant impact on the model`s retrieval performance and cut storage requirements by up to 50{\%}. Our new model, Jina-ColBERT-v2, demonstrates strong performance across a range of English and multilingual retrieval tasks,"
}
