A ModernBERT-base model continually pretrained on Intellectual Property (IP) data in Spanish and English. This specialized encoder model is optimized for understanding IP-related texts.

Model Description
Training Data
Training Configuration
Training Results
Intended Uses
Evaluation
How to Use
Additional Information

Model Description

Attribute	Value
Base Model	BSC-LT/MrBERT
Architecture	FlexBERT (22 layers, 768 hidden, 12 heads)
Parameters	~149M
Vocabulary Size	256,000 tokens
Max Sequence Length	8,192 tokens
Languages	Spanish (es), English (en)
Domain	Intellectual Property

Training Data

This model was trained on the gplsi/alia_intellectual_property dataset (Spanish and English subsets) and on gplsi/discriminative_counterfeit_es and gplsi/discriminative_counterfeit_en filtered to samples labeled "not fake". This work has also been partially supported by Project HEART-NLP (PID2024-156263OB-C22).

Data Processing Pipeline

Filtering: Selected only samples with language ∈ {"es", "en"} from gplsi/alia_intellectual_property and only samples labeled "not fake" from the counterfeit datasets.
Tokenization: BPE tokenization with MrBERT tokenizer (256k vocab)
Chunking: Packed into 8,192-token sequences
Split: 90% train / 10% validation

Training Configuration

Parameter	Value
Training Epochs	3
Sequence Length	8,192
MLM Probability	30% (train), 15% (eval)
Batch Size	32
Learning Rate	5e-5 (cosine decay to 5e-6)
Warmup	101 batches (1%)
Optimizer	StableAdamW
Precision	bfloat16
Hardware	2× NVIDIA Blackwell 6000 Pro

Training Results

Epoch	Training Loss
1	5.30
2	4.50
Final	2.53

Intended Uses

Primary Use Cases

IP-focused NLP: NER, text classification, and sentiment analysis for intellectual property documents
Semantic Search: Retrieval and similarity search for brand and regulatory queries
Information Extraction: Extraction of brand-, regulation-, and IP-related entities
Multilingual Support: Processing Spanish and English IP texts

Out-of-Scope Uses

General-purpose language understanding outside IP domain
Languages other than Spanish and English
Text generation (this is an encoder-only model)

Users should evaluate the model's outputs for fairness and bias in their specific applications.

How to Use

Transformers

python

from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")
tokenizer = AutoTokenizer.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")

# Fill-mask example
text = "El acceso a este [MASK] no implica cesión ni transmisión de derechos de explotación sobre el mismo.."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Get predictions
import torch
mask_token_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_token_id = outputs.logits[0, mask_token_index].argmax(axis=-1)
print(tokenizer.decode(predicted_token_id))

For Embeddings

python

from transformers import AutoModel, AutoTokenizer
import torch

model = AutoModel.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")
tokenizer = AutoTokenizer.from_pretrained("gplsi/Aitana-intellectual-property-mb-encoder")

text = "El acceso a este material no implica cesión ni transmisión de derechos de explotación sobre el mismo."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

with torch.no_grad():
    outputs = model(**inputs)
    embeddings = outputs.last_hidden_state.mean(dim=1)  # Mean pooling

Evaluation

This model was evaluated using the GLUE and SuperGLUE benchmarks.

Suite	Task/Metric	Seeds	Scores by seed	Mean +/- std
GLUE	CoLA MCC	4	19: 7.27, 8364: 13.42, 717: 11.28, 10536: 17.63	12.40 +/- 4.32
GLUE	MNLI acc	1	19: 69.56	69.56
GLUE	MNLI-mm acc	1	19: 70.51	70.51
GLUE	MRPC F1	5	19: 69.50, 8364: 71.27, 717: 73.02, 10536: 71.22, 90166: 70.30	71.06 +/- 1.32
GLUE	QNLI acc	1	19: 70.58	70.58
GLUE	QQP F1	1	19: 80.90	80.90
GLUE	RTE acc	5	19: 50.18, 8364: 50.54, 717: 48.74, 10536: 50.54, 90166: 48.74	49.75 +/- 0.93
GLUE	SST-2 acc	3	19: 83.72, 8364: 84.40, 717: 84.40	84.17 +/- 0.39
GLUE	STS-B Spearman	5	19: 43.16, 8364: 36.60, 717: 43.35, 10536: 41.89, 90166: 40.68	41.14 +/- 2.75
SuperGLUE partial	MNLI acc	1	19: 70.31	70.31
SuperGLUE partial	MNLI-mm acc	1	19: 70.36	70.36
SuperGLUE partial	RTE acc	5	19: 51.26, 8364: 50.54, 717: 48.74, 10536: 50.54, 90166: 48.38	49.89 +/- 1.26
SuperGLUE partial	BoolQ acc	3	23: 65.63, 42: 65.60, 6033: 64.62	65.28 +/- 0.57
SuperGLUE partial	CB acc	3	23: 67.86, 42: 66.07, 6033: 67.86	67.26 +/- 1.03
SuperGLUE partial	CB F1	3	23: 57.63, 42: 55.20, 6033: 47.21	53.35 +/- 5.45
SuperGLUE partial	COPA acc	5	23: 56.00, 42: 51.00, 6033: 51.00, 1337: 56.00, 24: 55.00	53.80 +/- 2.59
SuperGLUE partial	SWAG acc	1	19: 33.47	33.47
SuperGLUE partial	WiC acc	3	23: 56.27, 42: 57.99, 6033: 58.93	57.73 +/- 1.35

Additional Information

Author

The model has been developed by the Language and Information Systems Group (GPLSI) and the Centro de Inteligencia Digital (CENID), both part of the University of Alicante (UA), as part of their ongoing research in Natural Language Processing (NLP).

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública, co-financed by the EU – NextGenerationEU, within the framework of the project Desarrollo de Modelos ALIA.

Acknowledgments

We would like to express our gratitude to all individuals and institutions that have contributed to the development of this work.

Special thanks to:

We also acknowledge the financial, technical, and scientific support of the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA, whose contribution has been essential to the completion of this research.

License

This model is released under the Apache License 2.0.

Disclaimer

This model is intended for general purposes and is available under a permissive Apache License 2.0. Be aware that the model may have biases and/or undesirable outputs. Users deploying systems based on this model are responsible for mitigating risks and complying with applicable AI regulations.

Reference

If you use this model, please cite:

bibtex

@misc{modernbert-ip-2026-encoder,
  author = {Sepúlveda-Torres, Robiert and Estevanell-Valladares, Ernesto L. and Galeano, Santiago and Martínez-Murillo, Iván and Grande, Eduardo and Canal-Esteve, Miquel and Miró Maestre, María and Bonora, Mar and Gutierrez, Yoan and Abreu Salas, José Ignacio and Consuegra-Ayala, Juan Pablo and Lloret, Elena and Montoyo, Andrés and Muñoz-Guillena and Palomar, Manuel},
  title = {Aitana IP Encoder: Domain-Adapted Language Model for Spanish and English Intellectual Property},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/gplsi/Aitana-intellectual-property-mb-encoder}}
}

gplsi/Aitana-intellectual-property-mb-encoder

Aitana-intellectual-property-mb-encoder (Spanish & English)

Table of Contents