Architecture: ModernBERT (Answer.AI/LightOn)
Layers: 22
Hidden size: 1024
Attention heads: 16
Intermediate size: 1536
Vocab size: 32,128
Parameters: ~230M
Tokenizer: yhavinga/dutch-llama-tokenizer (SentencePiece, Dutch-optimized)

Training

Dataset: FineWeb2 Dutch
Steps: 2,000,000
Precision: bfloat16
Framework: JAX/Flax

Usage

python

from transformers import AutoTokenizer, ModernBertForMaskedLM

model = ModernBertForMaskedLM.from_pretrained("yhavinga/dmbert-1024h-22l-fineweb2-2000000")
tokenizer = AutoTokenizer.from_pretrained("yhavinga/dmbert-1024h-22l-fineweb2-2000000")

# Masked language modeling
inputs = tokenizer("Amsterdam is de <mask> van Nederland.", return_tensors="pt")
outputs = model(**inputs)

Citation

bibtex

@model{dmbert_1024h_fineweb2,
  title={Dutch ModernBERT 1024h-22L FineWeb2},
  author={Yeb Havinga},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/yhavinga/dmbert-1024h-22l-fineweb2-2000000}
}