diabase-reranker-1
Research preview. Training and corrected v2 bootstrap evaluation are complete.
Independent Swedish MTEB evaluation is mixed: the reranker degrades SweFAQ and does
not provide a consistent improvement on SweDN. It is published for transparent
research and evaluation, not as the recommended production reranker.
A Swedish-first European cross-encoder for precise document reranking.
diabase-reranker-1 scores a query and document together. It is designed to rerank a
small candidate set retrieved by diabase-embedding-1, improving precision where dense
embedding similarity alone cannot distinguish subtle relevance differences.
The model shares the Swedish-adapted EuroBERT-210m foundation used by
diabase-embedding-1, but it has separate weights and a separate relevance objective.
| |
|---|
| Status | Research preview; not recommended for production |
| Architecture | EuroBERT cross-encoder + scalar relevance head |
| Parameters | Approximately 210M |
| Primary language | Swedish |
| Secondary language | English replay is retained during adaptation |
| Maximum pair length | 512 tokens |
| Input | Query/document pair |
| Output | Relative relevance score |
| Recommended candidate count | Rerank approximately 20–50 retrieved documents |
| Companion retriever | Diabase/embedding-1 |
Quick start
1from sentence_transformers import CrossEncoder
2
3model = CrossEncoder("Diabase/reranker-1")
4
5query = "Vilka krav gäller för leverantörer av högrisksystem?"
6documents = [
7 "Leverantörer av högrisksystem måste arbeta med riskhantering, dokumentation, "
8 "datakvalitet och mänsklig tillsyn.",
9 "Stockholm är Sveriges huvudstad.",
10]
11
12pairs = [(query, document) for document in documents]
13scores = model.predict(pairs)
14
15ranked = sorted(zip(scores, documents), reverse=True)
16for score, document in ranked:
17 print(float(score), document)
Scores are intended for ranking documents for the same query. Do not assume scores
are calibrated probabilities or directly comparable across unrelated queries.
Intended use
- Reranking candidates from
diabase-embedding-1
- Improving precision in Swedish RAG systems
- Distinguishing closely related legal, technical and enterprise passages
- Selecting a small grounded context set before text generation
- Local and on-prem deployments where query/document data must remain private
Out of scope
- Searching an entire large corpus directly
- Treating relevance as factual correctness, authority or legal validity
- Using scores as calibrated probabilities
- Fully autonomous high-stakes decisions
- Generating answers or summaries
- Replacing access control or privacy enforcement
Why a separate reranker?
An embedding model encodes queries and documents independently, which makes retrieval
fast enough for large indexes. A cross-encoder reads each query/document pair jointly,
allowing richer token-level interaction but making it too expensive for exhaustive
search.
The intended system is:
1large document corpus
2 -> diabase-embedding-1 retrieves top 20–50
3 -> diabase-reranker-1 reorders those candidates
4 -> top 3–5 documents are passed to the generator
The reranker is optional. It will only be released if measured precision gains justify
its additional latency and operational complexity.
Model design
- Start from the Swedish-adapted EuroBERT-210m encoder created for the Diabase retrieval
stack.
- Add a scalar sequence-classification/relevance head.
- Train on query/positive pairs and difficult negatives.
- Optimize ordering quality, not text generation or embedding geometry.
Unlike diabase-embedding-1, the reranker does not produce reusable document vectors.
Every query/document pair requires a forward pass.
Training data
Training examples contain:
- A Swedish query
- One verified relevant document
- Multiple hard negatives retrieved by
diabase-embedding-1
- Optional graded relevance labels where review supports them
Hard negatives are documents that appear semantically plausible but do not answer the
query. Random unrelated negatives are insufficient because they make the task too easy
and do not teach the model the distinctions needed in production.
False-negative control
Retrieved negatives may actually be relevant even when they are not the originally
labelled positive passage. Before training:
- Remove exact and near-duplicate positives
- Check whether a candidate contains the gold answer
- Apply heuristic filtering
- Human-review a representative sample and high-impact ambiguous cases
Data provenance
The initial positives come from project-generated Swedish extractive-QA data. Candidate
documents and hard negatives come from the authorized retrieval corpus. The corrected
v2 pipeline groups semantic/topic families before splitting and excludes candidates
from the query's own group, known positives, answer-containing passages and near
duplicates.
The final training set contains 2,702 queries and 13,510 labelled pairs: one positive
and four mined negatives per retained query. The development set contains 516 queries
and 2,580 labelled pairs. Queries without four safe negatives in the top search window
were skipped rather than filled with questionable labels.
Evaluation
The evaluation compares:
- Ordering returned directly by
diabase-embedding-1
- The same candidate lists after
diabase-reranker-1
The held-out v2 test contains 455 queries and 455 documents across 32 test topic
groups. The reranker reorders the top 20 candidates from the v2 embedding model.
| Metric | Embedding order | Reranked | Absolute change |
|---|
| Recall@1 | 49.67% | 51.43% | +1.76 points |
| Recall@5 | 77.36% | 79.56% | +2.20 points |
| Recall@10 | 87.03% | 89.23% | +2.20 points |
| Recall@20 | 95.38% | 95.38% | unchanged |
| MRR@10 | 61.63% | 63.50% | +1.88 points |
| nDCG@10 | 67.71% | 69.68% | +1.97 points |
| Median relevant-document rank | 2 | 1 | -1 |
| Mean relevant-document rank | 4.26 | 3.88 | -0.38 |
On the development RTX 5070, scoring 9,100 pairs took 17.8 seconds: approximately
512 pairs/second or 39 ms per query when reranking 20 candidates. This is one local
throughput measurement, not a general latency guarantee.
The test queries and documents must be separate from embedding training, hard-negative
mining supervision and reranker training.
Independent Swedish MTEB reranking
The cross-encoder reranks the top 20 candidates from diabase-embedding-1. Candidate
Recall@20 is therefore fixed. The evaluation uses the same MTEB dataset revisions and
512-token limit as the embedding evaluation.
| Task and metric | Embedding order | Reranked | Change |
|---|
| SweFAQ Recall@1 | 30.21% | 22.22% | -7.99 points |
| SweFAQ MRR@10 | 41.15% | 35.54% | -5.60 points |
| SweFAQ nDCG@10 | 47.16% | 42.86% | -4.30 points |
| SweDN Recall@1 | 16.16% | 13.77% | -2.39 points |
| SweDN MRR@10 | 39.99% | 38.43% | -1.55 points |
| SweDN nDCG@10 | 34.02% | 34.18% | +0.15 points |
This fails the stable release gate. The model should not replace embedding order in a
production pipeline without domain-specific evaluation and additional training data.
Local latency
For 20 candidates per query:
- CPU: 1,455 ms mean, 1,713 ms p95 and 13.7 pairs/second.
- NVIDIA RTX 5070: 35.5 ms mean, 41.0 ms p95 and 563.3 pairs/second.
- Combined embedding/reranker benchmark peak GPU allocation: approximately 1.67 GB.
Measured with PyTorch 2.11.0 over 20 timed iterations. These are local measurements,
not service-level guarantees.
Release gate
The corrected bootstrap passes the internal quality gate: top-1 relevance, nDCG@10 and
MRR@10 improve while candidate-set Recall@20 is preserved. A stable release still
requires that it:
- Improves nDCG@10 and top-1 relevance over embedding order by a practically meaningful
margin
- Does not remove relevant documents from the final context set too often
- Meets the documented latency target on local hardware
- Shows gains on a separate human-reviewed Swedish benchmark
If those conditions are not met, Diabase will recommend using
diabase-embedding-1 without a reranker.
Limitations
- Cross-encoder inference is substantially slower than vector similarity.
- The model is Swedish-first; other-language ranking quality may be lower.
- Pair truncation can remove decisive evidence from long documents.
- Hard-negative quality strongly controls model quality.
- A relevance model can prefer persuasive or lexically similar misinformation.
- Scores may shift across domains and should not be treated as globally calibrated.
- Reranking cannot recover a relevant document that the embedding model never retrieved.
Security and privacy
Local inference keeps query/document text under the deployer's control, but callers must
still enforce authorization before scoring documents. Reranking does not neutralize
prompt injection, malicious documents, poisoned corpora or confidential-data leakage.
Relationship to other Diabase models
diabase-embedding-1: fast first-stage candidate retrievaldiabase-reranker-1: precise second-stage candidate ordering- Diabase Europa: grounded response generation from the selected context
The models are separate artifacts because they optimize different objectives and have
different latency profiles.
License and attribution
The Diabase derivative is released under Apache-2.0, consistent with the EuroBERT
foundation. Release weight SHA-256:
36ab2529642ed5e3f1ef857015575480921f8c3eb469764b271e9b87b32737f5.
EuroBERT citation:
1@misc{boizard2025eurobertscalingmultilingualencoders,
2 title={EuroBERT: Scaling Multilingual Encoders for European Languages},
3 author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves
4 and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot
5 and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan
6 and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse
7 and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes
8 and Ricardo Rei and Pierre Colombo},
9 year={2025},
10 eprint={2503.05500},
11 archivePrefix={arXiv},
12 primaryClass={cs.CL},
13 url={https://arxiv.org/abs/2503.05500}
14}
Maintainer
