This model is part of our reimplementation of the RoBERTa model, trained on Wikipedia and the Book Corpus only. We train this model for almost 100K steps, corresponding to 83 epochs. We provide the 84 checkpoints (including the randomly initialized weights before the training) to provide the ability to study the training dynamics of such models, and other possible use-cases.

These models were trained in part of a work that studies how simple statistics from data, such as co-occurrences affects model predictions, which are described in the paper Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions.

This is RoBERTa-base epoch_0.

This model was captured during a reproduction of RoBERTa-base, for English: it is a Transformers model pretrained on a large corpus of English data, using the Masked Language Modelling (MLM).

The intended uses, limitations, training data and training procedure for the fully trained model are similar to RoBERTa-base. Two major differences with the original model:

Using code from RoBERTa-base, here is an example based on PyTorch:

from transformers import pipeline

model = pipeline("fill-mask", model='yanaiela/roberta-base-epoch_83', device=-1, top_k=10)
model("Hello, I'm the <mask> RoBERTa-base language model")