This is the ELECTRA-Tiny language model that is effective for discriminative tasks with fewer than 6 million parameters.
@inproceedings{fields-kennington-2023-exploring,
title = "Exploring Transformers as Compact, Data-efficient Language Models",
author = "Fields, Clayton and
Kennington, Casey",
editor = "Jiang, Jing and
Reitter, David and
Deng, Shumin",
booktitle = "Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "
https://aclanthology.org/2023.conll-1.35",
doi = "10.18653/v1/2023.conll-1.35",
pages = "521--531",
abstract = "Large scale transformer models, trained with massive datasets have become the standard in natural language processing. The huge size of most transformers make research with these models impossible for those with limited computational resources. Additionally, the enormous pretraining data requirements of transformers exclude pretraining them with many smaller datasets that might provide enlightening results. In this study, we show that transformers can be significantly reduced in size, with as few as 5.7 million parameters, and still retain most of their downstream capability. Further we show that transformer models can retain comparable results when trained on human-scale datasets, as few as 5 million words of pretraining data. Overall, the results of our study suggest transformers function well as compact, data efficient language models and that complex model compression methods, such as model distillation are not necessarily superior to pretraining reduced size transformer models from scratch.",
}
This material is based upon work supported by the National Science Foundation under Grant No. 2140642.
