ModernBERT-base-mlm-wiki-code is a continued pre-trained version of answerdotai/ModernBERT-base, further trained on a large mixed corpus of English natural language text and multi-language source code using Masked Language Modeling (MLM).

The model was trained with a challenging 30% masking probability (vs. the standard 15% in BERT) over a 2048 token context window, achieving a final perplexity of 1.9507 — indicating strong and confident language + code understanding across both domains.

Evaluation Results

Evaluation was performed separately on each domain to understand per-domain performance.

Dataset	Eval Loss	Perplexity
WikiText (Natural Language)	1.3994	4.0526
The Stack V2 (Code)	0.4091	1.5054
Combined (NL + Code)	0.6728	1.9598

MLM Probability: 30% — Context Length: 2048 tokens

Model Details

Property	Value
Base Model	`answerdotai/ModernBERT-base`
Model Type	Masked Language Model (MLM)
Architecture	ModernBERT (Encoder-only Transformer)
Context Length	2048 tokens
MLM Probability	30%
Languages	English, C++, Go, Java, JavaScript, Python
License	MiT

Usage

Installation

pip install transformers torch

Load the Model

python

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
model = AutoModelForMaskedLM.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
model.config.reference_compile = False

Fill-Mask — Natural Language

python

from transformers import pipeline

pipe = pipeline("fill-mask", model="kd13/ModernBERT-base-mlm-wiki-code")

result = pipe("The capital of France is [MASK].")
for r in result:
    print(f"{r['token_str']:15s} → {r['score']:.4f}")

Fill-Mask — Source Code

python

from transformers import pipeline

pipe = pipeline("fill-mask", model="kd13/ModernBERT-base-mlm-wiki-code")

result = pipe("def fibonacci(n): return n if n <= 1 else fibonacci(n-1) [MASK] fibonacci(n-2)")
for r in result:
    print(f"{r['token_str']:15s} → {r['score']:.4f}")

Feature Extraction (Embeddings)

python

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")
model = AutoModel.from_pretrained("kd13/ModernBERT-base-mlm-wiki-code")

text = "def quicksort(arr): return arr if len(arr) <= 1 else ..."
inputs = tokenizer(text, return_tensors="pt", max_length=2048, truncation=True)

with torch.no_grad():
    outputs = model(**inputs)

# CLS token embedding — shape: (1, 768)
embedding = outputs.last_hidden_state[:, 0, :]
print(embedding.shape)

Citation

If you use this model, please cite the original ModernBERT paper:

bibtex

@article{modernbert2024,
  title   = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author  = {Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
  year    = {2024},
  url     = {https://arxiv.org/abs/2412.13663}
}

Acknowledgements

Base model by Answer.AI
Training data from WikiText and BigCode The Stack V2