This is a more up-to-date version of the
original tiny BERT referenced in
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models (English only, uncased, trained with WordPiece masking).
In addition to being more up-to-date, it is more CPU friendly than its base version, but its first version and is not perfect by no means. Took a day and 8x A100s to train. 🤗
The model was trained on a cleaned December 2022 snapshot of Common Crawl and Wikipedia.
This model was intended to be part of the OLM project, which has the goal of continuously training and releasing models that are up-to-date and comparable in standard language model performance to their static counterparts.
This is important because we want our models to know about events like COVID or
a presidential election right after they happen.
You can use the raw model for masked language modeling, but it's mostly intended to
be fine-tuned on a downstream task, such as sequence classification, token classification or question answering.
It looks like the olm tinybert is underperforming the original from a quick glue finetuning and dev evaluation:
1{'cola_mcc': 0.0, 'sst2_acc': 0.7981651376146789, 'mrpc_acc': 0.6838235294117647, 'mrpc_f1': 0.8122270742358079, 'stsb_pear': 0.67208
22873279731, 'stsb_spear': 0.6933378278505834, 'qqp_acc': 0.7766420762598881, 'mnli_acc': 0.6542027508914926, 'mnli_acc_mm': 0.6670056
3956875509, 'qnli_acc': 0.774665934468241, 'rte_acc': 0.5776173285198556, 'wnli_acc': 0.49295774647887325}
1{'cola_mcc': 0.0, 'sst2_acc': 0.7970183486238532, 'mrpc_acc': 0.6838235294117647, 'mrpc_f1': 0.8122270742358079, 'stsb_pear': -0.1597
28233085015087, 'stsb_spear': -0.13638650127051932, 'qqp_acc': 0.6292213609628794, 'mnli_acc': 0.5323484462557311, 'mnli_acc_mm': 0.54
365825874694874, 'qnli_acc': 0.6199890170236134, 'rte_acc': 0.5595667870036101, 'wnli_acc': 0.5352112676056338}
Probably messed up with hyperparameters and tokenizer a bit, unfortunately. Anyway Stay tuned for version 2 🚀🚀🚀
But please try it out on your downstream tasks, might be more performant. Should be cheap to fine-tune due to its size 🤗
The model and tokenizer were trained with this
December 2022 cleaned Common Crawl dataset plus this
December 2022 cleaned Wikipedia dataset.
The tokenized version of these concatenated datasets is
here.
The datasets were created with this
repo.
The model was trained according to the OLM BERT/RoBERTa instructions at this
repo.
