A Portuguese ModernBERT encoder model fine-tuned from
mmBERT with a custom Portuguese-optimized tokenizer.
NeoBERTugues is a Portuguese language model based on the ModernBERT architecture. It was created by:
-
Overlapping Tokens: For tokens that exist in both the new NeoBERTugues and mmBERT vocabulary, the original mmBERT embedding weights are preserved.
-
Old tokens: Tokens that exist in mmBERT tokenizer but not in the NeoBERTugues tokenizer were discarded.
-
New Token Initialization: For NeoBERTugues-specific tokens not present in mmBERT, embeddings are initialized using a statistical distribution matching approach:
- The embedding statistics (mean, standard deviation, skewness) of mmBERT's embeddings are computed
- A skewed normal distribution is fitted to the decoder bias values of overlapping tokens
- New token embeddings are sampled from distributions that match these statistics
- This ensures new tokens start in a statistically similar space to existing tokens, enabling faster convergence
This approach allows the model to leverage mmBERT's multilingual knowledge while adding Portuguese-specific vocabulary.
Unfortunately we were not able to run extensive benchmarks due to limited budget, so we're not claiming to have achieved SOTA results. Take these limited results with a grain of salt.
Evaluation using logistic regression probing on Portuguese NLP benchmarks (F1 Macro scores):
1from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline
2
3# Load model and tokenizer
4model = AutoModelForMaskedLM.from_pretrained("lorenzocc/NeoBERTugues")
5tokenizer = AutoTokenizer.from_pretrained("lorenzocc/NeoBERTugues")
6
7# Create fill-mask pipeline
8fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
9
10# Example usage
11result = fill_mask("O Brasil é um país <mask>.")
12for r in result:
13 print(f"{r['token_str']}: {r['score']:.1%}")
1import torch
2from transformers import AutoModel, AutoTokenizer
3
4model = AutoModel.from_pretrained("lorenzocc/NeoBERTugues")
5tokenizer = AutoTokenizer.from_pretrained("lorenzocc/NeoBERTugues")
6
7text = "NeoBERTugues e um modelo de linguagem para portugues."
8inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
9
10with torch.no_grad():
11 outputs = model(**inputs)
12
13# Get sentence embedding (mean pooling)
14attention_mask = inputs["attention_mask"]
15last_hidden = outputs.last_hidden_state
16masked_hidden = last_hidden * attention_mask.unsqueeze(-1)
17sentence_embedding = masked_hidden.sum(dim=1) / attention_mask.sum(dim=1, keepdim=True)
1@misc{cesconetto2026neobertugues,
2 author = {Cesconetto, Lorenzo},
3 title = {NeoBERTugues: A Portuguese ModernBERT Model},
4 year = {2026},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/lorenzocc/NeoBERTugues}
7}
Special thanks to
CloudWalk for making this possible through their AI Residency Program.
