SIGMA-BERT is a Slovenian encoder language model based on the ModernBERT architecture. It was pretrained from scratch using the masked language modeling (MLM) objective on a diverse collection of Slovenian text, including web, news, parliamentary proceedings, scientific publications, and user-generated content. The model supports sequence lengths of up to 8192 tokens and is intended for downstream language understanding tasks such as classification, retrieval, semantic similarity, and token-level prediction.
As a pretrained encoder, SIGMA-BERT is intended to be fine-tuned for downstream Slovenian language understanding tasks such as:
Text classification (e.g. topic, sentiment, genre)
Named entity recognition and other token classification tasks
Extractive question answering
Natural language inference / semantic textual similarity
Dense retrieval and semantic search, including over long documents
As a backbone for rerankers in retrieval-augmented generation (RAG) pipelines
It is available in the following variants:
SIGMA-BERT-short - 22 layers, 149 million parameters, 1,024 token context
SIGMA-BERT-base - 22 layers, 149 million parameters, 8,192 token context
SIGMA-BERT-large - 28 layers, 395 million parameters, 8,192 token context
Model Details
SigmaBERT-base is built on the ModernBERT architecture and pretrained on Slovenian text with a Masked Language Modeling (MLM) objective.
Property
Value
Language
Slovenian
Training objective
Masked Language Modeling (MLM)
Parameters
149M
Layers
22
Hidden size
768
Intermediate size (GeGLU)
1,152
Attention heads
12
Vocab size
50,000
Max sequence length
8,192 (extended from 1,024)
Attention pattern
Global attention every 3rd layer; local sliding-window attention (128 tokens) elsewhere
Activation
GeGLU
Normalization / linear layers
Pre-LayerNorm, no bias terms
Training
SIGMA-BERT was pretrained in two stages.
Stage 1
The initial pretraining stage focused on learning general Slovenian language representations.
Maximum sequence length: 1,024
MLM masking probability: 30%
Learning rate: 5e-4
Optimizer: StableAdamW
Weight decay: 0.1
Stage 2
The second stage extended the context window and continued pretraining on long-form, information-dense text.
Maximum sequence length: 8,192
MLM masking probability: 15%
Learning rate: 5e-5
Optimizer: StableAdamW
Weight decay: 0.1
Increased RoPE base frequency to support the longer context window
Both stages used a warmup–plateau–cooldown learning rate schedule.
Training Data
The training corpus contains approximately 13.88 billion tokens of Slovenian text.
Stage 1 used a mixture of publicly available Slovenian corpora:
OpenScience — Slovenian academic and scientific texts
siParl 4.0 — Slovenian parliamentary proceedings corpus
MaCoCu-sl 2.0 — Slovenian web corpus
Stage 2 continued pretraining using:
OpenScience
siParl 4.0
Tokenizer
SIGMA-BERT uses a custom SentencePiece BPE tokenizer trained from scratch with a vocabulary of 50,000 tokens. The tokenizer employs Metaspace pre-tokenization together with newline isolation and digit splitting to better preserve document structure and numeric information.
⚠️ Note on [MASK] spacing: SIGMA-BERT's tokenizer uses a Metaspace pre-tokenizer, which encodes the space before a word as part of that word's token (e.g. " mesto" → ▁mesto). Because of this, if you write [MASK] in a text string, it should be placed directly against the preceding word, with no space ("je[MASK] mesto", not "je [MASK] mesto"). Adding a literal space before [MASK] produces an extra, out-of-distribution ▁ token that the model never saw during training:
