Views
No views yet
jhu-clsp/ettin-encoder-68m. It was trained on Ms-Marco using loss distillRankNET as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-68m-DistillRankNET")
5model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-68m-DistillRankNET")
6
7features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)1000 documents retrieved by naver/splade-v3-distilbert.| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 34.13 | 40.46 |
| trec2019 | 98.84 | 74.07 |
| trec2020 | 91.51 | 71.92 |
| fever | 74.80 | 75.23 |
| arguana | 14.35 | 21.10 |
| climate_fever | 16.46 | 12.19 |
| dbpedia | 71.85 | 42.73 |
| fiqa | 41.85 | 34.25 |
| hotpotqa | 84.44 | 66.11 |
| nfcorpus | 53.60 | 32.11 |
| nq | 48.74 | 53.78 |
| quora | 76.33 | 78.23 |
| scidocs | 23.82 | 12.96 |
| scifact | 58.10 | 60.19 |
| touche | 60.96 | 35.49 |
| trec_covid | 91.02 | 75.07 |
| robust04 | 66.32 | 42.11 |
| lotte_writing | 70.84 | 61.51 |
| lotte_recreation | 57.85 | 52.52 |
| lotte_science | 48.94 | 40.54 |
| lotte_technology | 52.33 | 43.25 |
| lotte_lifestyle | 69.70 | 60.54 |
| Mean In Domain | 74.83 | 62.15 |
| BEIR 13 | 55.10 | 46.11 |
| LoTTE (OOD) | 61.00 | 50.08 |