A MiniLM-L12 cross-encoder passage reranker that takes the normalised BM25 score as a text token in its input, reproducing the CE_BM25CAT method of Askari et al., "Injecting the BM25 Score as Text Improves BERT-Based Re-rankers", ECIR 2023.

Its matched baseline — same data, same schedule, no injection — is Amdestya/cat-minilm-l12.

Input format — read before using

python

text_a = f"{bm25_score} [SEP] {query}"     # score BEFORE the query
text_b = passage
# tokenises to: [CLS] score [SEP] query [SEP] passage [SEP]

⚠️ This ordering differs from the paper's Eq. 3, which places the score between query and passage. This model follows the authors' released training code, which puts it before the query. A cross-encoder fed a layout it never saw in training returns confident nonsense rather than an error, so the ordering matters.

The score must be normalised the same way

bm25_score = int(((raw_bm25 - 0) / (50 - 0)) * 100) # Min-Max, GLOBAL setting, then x100 as an integer

Min-Max normalisation in the global setting — fixed constants min=0, max=50, not per-query statistics — then scaled by 100 and truncated to an integer, per the paper's §3.3. Raw or per-query normalised scores put the feature on a distribution the model has never seen.

Training scores came from pyserini / Lucene BM25Similarity(k1=0.82, b=0.68) (the Anserini-tuned MS MARCO settings). Other BM25 implementations — Terrier's defaults are k1=1.2, b=0.75 — produce a different score distribution, which shifts the injected integers.

Usage

python

from sentence_transformers import CrossEncoder

model = CrossEncoder("Amdestya/bm25cat-minilm-l12", max_length=256)

def norm(raw):
    return int((raw / 50) * 100)

query   = "what is a cat"
passage = "A cat is a small domesticated carnivorous mammal."
raw_bm25 = 36.5

score = model.predict([[f"{norm(raw_bm25)} [SEP] {query}", passage]])

Verify the tokenisation rather than assuming the literal [SEP] resolves as intended:

python

print(model.tokenizer.convert_ids_to_tokens(
    model.tokenizer(f"73 [SEP] {query}", passage)["input_ids"]))
# ['[CLS]', '73', '[SEP]', 'what', 'is', 'a', 'cat', '[SEP]', 'a', 'cat', ...]

Three [SEP] tokens, with the score as its own token after [CLS]. Outputs are raw logits (identity activation), unbounded, higher = more relevant.

Training


base model	`microsoft/MiniLM-L12-H384-uncased`
objective	MSE against a BERT_CAT teacher ensemble (Hofstätter et al.)
training pairs	5,000,000 of 79,518,568 available
steps	156,250 @ batch 32, 1 epoch
optimizer	Adam, lr 7e-6, 5,000 warmup steps
max length	256
hardware	1 × RTX A5000

This is a knowledge-distillation setup, following the authors' released notebook, rather than the cross-entropy objective described in the paper's §4. Worth noting when interpreting results: the teacher ensemble's scores are computed without any BM25 input, so the training objective does not directly reward the model for using the injected feature.

Intended use

Built as a reference artifact for a dissertation study on reproducing IR papers with LLMs.