This is a multilingual cross-encoder model. It performs cross-attention between a question-passage pair and outputs a relevance score between 0 and 1. The model should be used as a reranker for semantic search: given a query, encode the latter with some candidate passages -- e.g., retrieved with BM25 or a bi-encoder -- then sort the passages in a decreasing order of relevance according to the model's predictions. The model uses an XMOD backbone, which allows it to learn from monolingual fine-tuning in a high-resource language, like English, and performs zero-shot transfer to other languages.

Here are some examples for using the model with Sentence-Transformers, FlagEmbedding, or Huggingface Transformers.

Start by installing the library: pip install -U sentence-transformers. Then, you can use the model like this:

Start by installing the library: pip install -U FlagEmbedding. Then, you can use the model like this:

Start by installing the library: pip install -U transformers. Then, you can use the model like this:

[to come...]

We use the English training samples from the MS MARCO passage ranking dataset, which contains 8.8M passages and 539K training queries. We use the BM25 negatives provided by the official dataset and sample 1M (q, p) pairs with a 1/4 positive-to-negative ratio (i.e., 250k query-positive pairs for 750k query-negative pairs).

The model is initialized from the xmod-base checkpoint and optimized via the binary cross-entropy loss (as in monoBERT). It is fine-tuned on one 32GB NVIDIA V100 GPU for 5 epochs using the AdamW optimizer with a batch size of 32, a peak learning rate of 2e-5 with warm up along the first 10% of training steps and linear scheduling. We set the maximum sequence lengths for the concatenated question-passage pairs to 512 tokens.