This model has only English and Russian tokens left in the vocabulary. Thus making it twice as small as the original model while producing the same embeddings.
This model was trained on the
MMARCO dataset. It is a machine translated version of MS MARCO using Google Translate. It was translated to 14 languages. In our experiments, we observed that it performs also well for other languages.
As a base model, we used the
multilingual MiniLMv2 model.
The model can be used for Information Retrieval: Given a query, encode the query will all possible passages (e.g. retrieved with ElasticSearch). Then sort the passages in a decreasing order. See
SBERT.net Retrieve & Re-rank for more details. The training code is available here:
SBERT.net Training MS MarcoThe usage becomes easy when you have
SentenceTransformers installed. Then, you can use the pre-trained models like this:
1from sentence_transformers import CrossEncoder
2model = CrossEncoder('model_name')
3scores = model.predict([('Query', 'Paragraph1'), ('Query', 'Paragraph2') , ('Query', 'Paragraph3')])
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4model = AutoModelForSequenceClassification.from_pretrained('model_name')
5tokenizer = AutoTokenizer.from_pretrained('model_name')
6
7features = tokenizer(['How many people live in Berlin?', 'How many people live in Berlin?'], ['Berlin has a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.', 'New York City is famous for the Metropolitan Museum of Art.'], padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)
The model has been truncated in
this notebook.
