ALIA MrBERT Spanish Legal and Administrative Reranker Model
This repository contains ALIA MrBERT Spanish Legal and Administrative Reranker, a Spanish legal domain cross-encoder (reranker) model for information retrieval and document ranking tasks. It is built upon MrBERT-es, a bilingual (Spanish–English) foundational language model based on the ModernBERT architecture, and fine-tuned on domain-specific legal and administrative data using a Curriculum Learning strategy.
[!WARNING]
DISCLAIMER: This model is a domain-specific proof-of-concept designed to demonstrate retrieval capabilities in the Spanish legal and administrative domain.
While optimized for this domain, results should be verified against official legal sources. The model may fail in out-of-domain or adversarial inputs.
🎓 Training strategy: Curriculum Learning (easy → medium → hard)
⚙️ Negative mining: Positive-Aware Hard Negative Mining
Architecture
This model uses the same base architecture as MrBERT-es, formatted as a Cross-Encoder for sentence pair classification:
Base Architecture
ModernBERT
Total Parameters
~150M
Hidden size
768
Intermediate size
1,152
Attention heads
12
Hidden layers
22
Context length
8,192 tokens
Vocabulary size
51,200
Precision
bfloat16
Model Type
Cross-Encoder
Training
Training Strategy: Curriculum Learning
The model was fine-tuned using a Curriculum Learning strategy, progressively increasing the difficulty of training examples. For this Cross-Encoder, the training focused on the most challenging examples: pairs that a Bi-Encoder might struggle to distinguish.
The dataset consists of text pairs incorporating hard negatives mined from the corpus SINAI/ALIA-es-legal-administrative-triplets/train. For the Cross-Encoder, the data is flattened into independent {query, document, label} pairs, where label is either 1.0 (relevant) or 0.0 (irrelevant).
Refinement with Hard Negatives:
Training uses mined hard negatives to force the model to distinguish fine-grained nuances. Examples are processed in order of increasing difficulty (easy → medium → hard) to improve convergence and generalization.
Hyperparameter Optimization
Before training, hyperparameter search was conducted using Optuna (20 trials) to maximize NDCG@10 (with fallback to MRR@10) on a control subset:
This model is designed for document reranking and semantic matching tasks in the Spanish legal and administrative domain. Primary use cases include:
RAG pipelines: Reranking retrieved context chunks for language models
Search pipelines: Improving initial retrieval (e.g. BM25 or Bi-encoder) by doing precise cross-encoding over the top-k results
Legal text matching: Determining high-resolution entailment or relevance between queries and legal clauses
Out-of-Scope Use
General-domain retrieval (the model is specialized for legal/administrative Spanish)
Fast, large-scale search across millions of documents (use a Bi-encoder first, then rerank the top-k results with this Cross-encoder)
Cross-lingual retrieval beyond Spanish
How to Use
With sentence-transformers
python
1from sentence_transformers import CrossEncoder
23model = CrossEncoder("SINAI/ALIA-MrBERT-es-legal-administrative-reranker")45query ="¿Cuáles son los requisitos para solicitar una prestación por desempleo?"6documents =[7"El trabajador que cese en su actividad laboral tendrá derecho a la prestación por desempleo...",8"La prestación por desempleo contributiva se reconoce a quienes hayan cotizado al menos 360 días...",9"El subsidio de incapacidad temporal requiere un certificado médico en vigor.",10]1112# We want to score the query with each document13pairs =[[query, doc]for doc in documents]1415scores = model.predict(pairs)16print(scores)
Evaluation
The model was evaluated using the MTEB (Massive Text Embedding Benchmark) framework, adapted for the legal domain. The main reported metric is NDCG@10 (Normalized Discounted Cumulative Gain at k=10), which is the standard metric used in retrieval leaderboards and aligns with the metric reported in the MrBERT family.
Note: The BGE reranker and Nemotron reranker are significantly larger scale models (billions of parameters), yet our domain-specific 150M parameter cross-encoder performs comparably or better on specialized legal Spanish datasets (e.g., ssld, pairs650, pairs1.3k).
Limitations and Biases
Known Limitations
Domain specificity: The model is highly optimized for Spanish legal texts. Its zero-shot capabilities on general domains are weaker compared to massive generalist rerankers.
Latency: Being a cross-encoder, it can be computationally heavy to score many document-query pairs. Use it to rerank a maximum of typical 20-100 top documents fetched by a fast bi-encoder.
Legal accuracy: Semantic similarity does not guarantee legal correctness.
Biases
The model reflects biases present in Spanish legislation, case law, and administrative corpora.
If you use this model in your research, please cite:
bibtex
1@misc{ALIA-MrBERT-es-legal-administrative-reranker,
2 title = {ALIA MrBERT Spanish Legal and Administrative Reranker Model},
3 author = {SINAI Research Group, Universidad de Jaén},
4 year = {2026},
5 publisher = {HuggingFace},
6 howpublished = {\url{https://huggingface.co/SINAI/ALIA-MrBERT-es-legal-administrative-reranker}}
7}
Please also cite the base model:
bibtex
1@misc{tamayo2026mrbertmodernmultilingualencoders,
2 title={MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation},
3 author={Daniel Tamayo and Iñaki Lacunza and Paula Rivera-Hidalgo and Severino Da Dalt and Javier Aula-Blasco and Aitor Gonzalez-Agirre and Marta Villegas},
4 year={2026},
5 eprint={2602.21379},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2602.21379},
9}
Funding
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ALIA.
Acknowledgments
This dataset has been generated thanks to CEATIC (Centro de Estudios Avanzados en Tecnologías de la Información y de la Comunicación) – UJA (Universidad de Jaén) which provided the needed computational resources on its clusters.
