You can deploy this model and start earning money today!

dicta-il/
neodictabert

Name: neodictabert
Availability: InStock

Views

No views yet

NeoDictaBERT: Pushing the Frontier of BERT models in Hebrew

This is the base model pretrained only on Hebrew. We recommend using the model pretrained on English and Hebrew here.

Following the success of ModernBERT and NeoBERT, we set out to train a Hebrew version of NeoBERT.

Introducing NeoDictaBERT: A Next-Generation BERT-style model trained specifically for Hebrew, technical report coming soon.

Supported Context Length: 4,096 (~3,200 words)

Trained on a total of 235B tokens (5 epochs) with a context length of 1,024, and another 50B tokens with a context length of 4,096.

Sample usage:

python

from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('dicta-il/neodictabert')
model = AutoModelForMaskedLM.from_pretrained('dicta-il/neodictabert', trust_remote_code=True)

model.eval()

sentence = 'בשנת 1948 השלים אפרים קישון את [MASK] בפיסול מתכת ובתולדות האמנות והחל לפרסם מאמרים הומוריסטיים'

output = model(tokenizer.encode(sentence, return_tensors='pt'))
# the [MASK] is the 7th token (including [CLS])
import torch
top_2 = torch.topk(output.logits[0, 7, :], 2)[1]
print('\n'.join(tokenizer.convert_ids_to_tokens(top_2))) # should print לימודיו / הכשרתו

Performance

Please see our technical report here for performance metrics. The model outperforms previous SOTA models on almost all benchmarks, with a noticeable jump in the QA scores which indicate a much deeper semantic understanding.

Citation

If you use NeoDictaBERT in your research, please cite NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew

BibTeX:

bibtex

@misc{shmidman2025neodictabertpushingfrontierbert,
      title={NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew}, 
      author={Shaltiel Shmidman and Avi Shmidman and Moshe Koppel},
      year={2025},
      eprint={2510.20386},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.20386}, 
}

License

Shield:

CC BY 4.0

This work is licensed under a Creative Commons Attribution 4.0 International License.