This model is a distilled version of the BERT base model. It was introduced in this paper. The code for the distillation process can be found here. This model is uncased: it does not make a difference between english and English.

DistilBERT is a transformers model, smaller and faster than BERT, which was pretrained on the same corpus in a self-supervised fashion, using the BERT base model as a teacher. This model is further finetuned on the DB_PEDIA Dataset which can be found here. This dataset consists of 342,782 Wikipedia articles that have been cleaned and classified into hierarchical classes. The classification system spans three levels, with 9 classes at the first level, 70 classes at the second level, and 219 classes at the third level.

You can use the model to extract structured content and organizing it into taxonomic categories.

The model outputs the classification according to the label number which can be mapped by the following lines in the code snippet:

labelint = ['LABEL_0', 'LABEL_1', 'LABEL_2', 'LABEL_3', 'LABEL_4', 'LABEL_5', 'LABEL_6', 'LABEL_7', 'LABEL_8']

labeltxt = np.loadtxt("TASK2/label_vals/l1.txt", dtype="str")

(where labeltxt is : Agent, Device, Event, Place, Species, SportsSeason, TopicalConcept, UnitOfWork, Work)

You can use this model directly with a pipeline:

Even if the training data used for this model could be characterized as fairly neutral, this model can have biased predictions. It also inherits some of the bias of its teacher model.