BERT model trained on a lemmatized corpus containing Buddhist and non-Buddhist Sanskrit texts.
Model description
The model has the bert architecture and was pretrained from scratch as a masked language model
on the lemmatized Sanskrit corpus. Due to lack of resources and to prevent overfitting, the model is smaller than bert-base,
i.e. the number of attention heads and hidden layers have been reduced to 8 and the context has been reduced to 128 tokens. Vocabulary size is 10000 tokens.
How to use it
model = AutoModelForMaskedLM.from_pretrained("Matej/bert-small-buddhist-nonbuddhist-sanskrit")
tokenizer = AutoTokenizer.from_pretrained("Matej/bert-small-buddhist-nonbuddhist-sanskrit", use_fast=True)
Intended uses & limitations
MIT license, no limitations
Training and evaluation data
See the paper 'Embeddings models for Buddhist Sanskrit' for details on the corpora and the evaluation procedure.
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 5e-05
train_batch_size: 32
eval_batch_size: 4
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
