Cartographer is a small, fast cross-encoder that scores how well a shopping query matches a product. It takes a (query, product) pair and returns a single relevance score for reranking a candidate list. It is a drop-in upgrade for the general-purpose reranker it is built on: same size, same speed, measurably better on product search.

It was fine-tuned from cross-encoder/ms-marco-MiniLM-L-6-v2 on Amazon's public Shopping Queries Dataset (ESCI). The base model was trained on web-search passages; product listings are not web text, so a domain-tuned model of the same 22.7M-parameter footprint does meaningfully better.

Author / Co-author:

Author : 2013khansohail (Sohail Khan) Co-Author : najibkhanlive (Najibur Rahman Khan)

Results

Evaluated on the held-out ESCI test split (US / English): 8,956 queries, 181,701 (query, product) pairs. Both models scored at max_length=256, identical settings : the comparison is apples-to-apples. The headline metric is nDCG (graded relevance), which is what the E/S/C/I labels are for.

Metric	Baseline (ms-marco-MiniLM-L6)	Cartographer	Δ abs	Δ rel
nDCG@5	0.6748	0.7272	+0.0524	+7.76%
nDCG@10	0.7193	0.7642	+0.0449	+6.24%
nDCG@20	0.8064	0.8354	+0.0290	+3.60%

The gain is largest at the top of the ranking (nDCG@5), which is where it matters most for product search : users see the first few results. There were no regressions: every reported cutoff improved.

Why it works: it learned the relevance gradation

ESCI labels relevance on four graded levels : Exact, Substitute, Complement, Irrelevant : and the model was trained to regress onto graded relevance, not a binary relevant/not flag. The effect is visible in how cleanly the model separates the four classes. Mean model score per true label on the test set:

True label	Baseline mean (± std)	Cartographer mean (± std)
E (Exact)	0.024 (± 6.14)	0.283 (± 1.27)
S (Substitute)	−2.228 (± 5.81)	−0.648 (± 1.02)
C (Complement)	−0.787 (± 6.09)	−0.781 (± 1.04)
I (Irrelevant)	−5.558 (± 5.93)	−1.140 (± 1.11)

The baseline's scores are noise-dominated : its per-class standard deviation (~6) is larger than the gaps between class means, so it barely separates the labels relative to its own spread. Cartographer collapses that noise to std ≈ 1 while producing a clean, monotonic E > S > C > I ordering. That tightening is the model learning to rank product relevance.

Intended use

A second-stage reranker: retrieve a candidate set of products with a fast first-stage system (BM25 or a bi-encoder), then rerank the top-k with Cartographer for sharper ordering. It is designed to be cheap to serve at query time, which is exactly why lightweight cross-encoders get deployed in production.

Usage

python

from sentence_transformers import CrossEncoder

model = CrossEncoder("2013khansohail/cartographer-ecommerce-reranker-MiniLM-L6-v2", max_length=256)

query = "wireless noise cancelling headphones"
products = [
    "Sony WH-1000XM5 Wireless Noise Cancelling Headphones",
    "Wired Earbuds with Microphone, 3.5mm",
    "Headphone Stand Aluminum Desk Holder",
    "Bluetooth Speaker Waterproof Portable",
]

scores = model.predict([(query, p) for p in products])
ranked = sorted(zip(products, scores), key=lambda x: x[1], reverse=True)
for product, score in ranked:
    print(f"{score:.3f}  {product}")

Training

Base model: cross-encoder/ms-marco-MiniLM-L-6-v2 (22.7M params). Fine-tuning from a reranker checkpoint rather than a raw backbone converges faster and higher on a fixed dataset.
Data: ESCI Shopping Queries, US/English, small_version split. Product text is the product title (title-only recipe for v1).
Objective: pointwise regression (MSE with a sigmoid activation) onto graded relevance gains mapped as E=1.0, S=0.1, C=0.01, I=0.0 : Amazon's ranking-task convention. The gains and the model's sigmoid-bounded output share the same [0, 1] scale.
Recipe: 2 epochs with early stopping on validation nDCG@10; best checkpoint loaded at the end. fp16, effective batch size 64 (16 × 4 gradient accumulation), lr 2e-5, on a single consumer GPU (NVIDIA RTX 3050, 4GB).

On training length (an honest finding): one epoch reaches ~99% of the final quality (val nDCG@10 ≈ 0.759). A second epoch with early stopping adds ~0.4 nDCG@10, the gain transfers cleanly to the sealed test set (not val overfitting), and it specifically resolves the Substitute-vs-Complement ordering that one epoch leaves tangled. So 2 epochs is the shipped recipe : but one epoch is a defensible, cheaper alternative.

Limitations

Amazon distribution. ESCI is Amazon data, so the model inherits Amazon's product distribution and category mix. It may need further tuning to transfer to other catalogs.
Substitute vs. Complement is the soft boundary. The model orders S above C directionally, but S and C remain the closest pair and their score distributions overlap (std ≈ 1, means ~0.13 apart). This is expected: the gain mapping places both near zero, so MSE has little incentive to push them apart. If distinguishing substitutes from complements is critical for your use case, a different gain mapping or a ranking loss is the place to look.
Lightweight ceiling. This is the best model in its weight class, not the best model available. Some hard queries will need a heavier reranker. The pitch is "drop-in upgrade at the same cost," not "state of the art."
English-only, title-only product text in this version.

Citation

Built on the Amazon Shopping Queries Dataset (ESCI):

Reddy et al., Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search (2022). https://github.com/amazon-science/esci-data

Base model: cross-encoder/ms-marco-MiniLM-L-6-v2.

Training code, data prep, and full evaluation (reproducible end to end): github.com/sohail000/cartographer-reranker