A modern variant of BioBERT based on ModernBERT. We continued the masked language modeling pre-training task for 1.000.000 steps on PubMed abstracts.

Batchsize: 512
Learningrate: 1e-4
Warmupsteps: 500
Learning Rate Scheduler: Cosine Schedule
Max. Sequence Length: 512
Precision: bfloat16

datasets: