Views
No views yet
jhu-clsp/ettin-encoder-32m. It was trained on Ms-Marco using loss ADR as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-32m-ADR-MSE")
5model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-32m-ADR-MSE")
6
7features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
8
9model.eval()
10with torch.no_grad():
11 scores = model(**features).logits
12 print(scores)1000 documents retrieved by naver/splade-v3-distilbert.| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 30.21 | 35.72 |
| trec2019 | 83.41 | 62.03 |
| trec2020 | 85.55 | 62.79 |
| fever | 71.84 | 72.44 |
| arguana | 7.77 | 11.69 |
| climate_fever | 21.49 | 15.47 |
| dbpedia | 63.40 | 34.50 |
| fiqa | 33.97 | 25.77 |
| hotpotqa | 68.51 | 50.93 |
| nfcorpus | 44.83 | 25.25 |
| nq | 37.55 | 42.18 |
| quora | 71.95 | 73.20 |
| scidocs | 17.86 | 9.43 |
| scifact | 48.13 | 50.36 |
| touche | 66.33 | 31.05 |
| trec_covid | 80.14 | 60.85 |
| robust04 | 57.89 | 33.16 |
| lotte_writing | 60.22 | 50.65 |
| lotte_recreation | 49.07 | 44.49 |
| lotte_science | 40.88 | 34.02 |
| lotte_technology | 41.87 | 34.09 |
| lotte_lifestyle | 59.49 | 50.96 |
| Mean In Domain | 66.39 | 53.51 |
| BEIR 13 | 48.75 | 38.70 |
| LoTTE (OOD) | 51.57 | 41.23 |