Property	Value
Architecture	ModernBERT + Mean Pooling + L2 Normalize
Parameters	110M
Hidden size	768
Layers	12
Attention heads	12
Vocab size	32,768
Max sequence	1,024 tokens
Embedding dim	768 (L2 normalized)

Training

Pretraining: Masked Language Modeling on domain-specific glossary corpus
Dataset: mjbommar/ogbert-v1-mlm - derived from OpenGloss, a synthetic encyclopedic dictionary with 537K senses across 150K lexemes
Checkpoint: Step 8K (selected for optimal downstream performance)
Key finding: L2 normalization of embeddings is critical for clustering/retrieval performance

Performance

Document Clustering (ARI)

Evaluated on 80 domain-specific documents across 10 categories using KMeans clustering.

Model	Params	ARI	Cluster Acc
OGBert-110M-Sentence	110M	0.941	0.975
BERT-base	110M	0.896	0.950
RoBERTa-base	125M	0.941	0.975
MiniLM-L6-v2	22M	0.833	0.925

OGBert-110M-Sentence matches RoBERTa-base on clustering and beats MiniLM by 13%.

Document Retrieval (MRR)

Mean Reciprocal Rank for same-category document retrieval.

Model	Params	Sample MRR
OGBert-110M-Sentence	110M	0.959
BERT-base	110M	0.994
RoBERTa-base	125M	0.989
MiniLM-L6-v2	22M	0.958

Word Similarity (SimLex-999)

Spearman correlation between model cosine similarities and human judgments.

Model	Params	SimLex-999 (ρ)
OGBert-110M-Sentence	110M	0.345
BERT-base	110M	0.070
RoBERTa-base	125M	-0.061

OGBert-110M-Sentence achieves 5x better word similarity than BERT-base.

Summary vs Baselines

At the same size as BERT-base, OGBert-110M-Sentence achieves:

5x better word similarity (SimLex)
Matches RoBERTa on clustering (ARI)
13% better clustering than MiniLM

Usage

Sentence-Transformers (Recommended)

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('mjbommar/ogbert-110m-sentence')
embeddings = model.encode(['your text here'])  # L2 normalized by default

Example - Domain Similarity:

python

sentences = [
    'The financial audit revealed discrepancies in the quarterly report.',
    'An accounting review found errors in the fiscal statement.',
    'The patient was diagnosed with acute respiratory infection.',
]
embeddings = model.encode(sentences)

The model correctly identifies higher similarity within the same domain.

Direct Transformers Usage

python

from transformers import AutoModel, AutoTokenizer
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained('mjbommar/ogbert-110m-sentence')
model = AutoModel.from_pretrained('mjbommar/ogbert-110m-sentence')

inputs = tokenizer('your text here', return_tensors='pt', padding=True, truncation=True)
outputs = model(**inputs)

# Mean pooling + L2 normalize (critical for performance)
mask = inputs['attention_mask'].unsqueeze(-1)
pooled = (outputs.last_hidden_state * mask).sum(1) / mask.sum(1)
embeddings = F.normalize(pooled, p=2, dim=1)

For Fill-Mask Tasks

Use mjbommar/ogbert-110m-base instead.

Citation

If you use this model, please cite the OpenGloss dataset:

bibtex

@article{bommarito2025opengloss,
  title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph},
  author={Bommarito II, Michael J.},
  journal={arXiv preprint arXiv:2511.18622},
  year={2025}
}

License

Apache 2.0