text: "يحكى أن مزارعا مخادعا قام ببيع بئر الماء الموجود في أرضه لجاره مقابل مبلغ كبير من المال"
text: "القدس مدينة تاريخية، بناها الكنعانيون في"
text: "كان يا ما كان في قديم الزمان"
Arabic GPT2
You can find more information in our paper AraGPT2
The code in this repository was used to train all GPT2 variants. The code support training and fine-tuning GPT2 on GPUs and TPUs via the TPUEstimator API.
GPT2-base and medium uses the code from the gpt2 folder and can trains models from the minimaxir/gpt-2-simple repository.
These models were trained using the lamb optimizer and follow the same architecture as gpt2 and are fully compatible with the transformers library.
GPT2-large and GPT2-mega were trained using the imcaspar/gpt2-ml library, and follow the grover architecture. You can use the pytorch classes found in grover/modeling_gpt2.py as a direct replacement for classes in the transformers library (it should support version v4.x from transformers).
Both models are trained using the adafactor optimizer, since the adam and lamb optimizer use too much memory causing the model to not even fit 1 batch on a TPU core.
AraGPT2 is trained on the same large Arabic Dataset as AraBERTv2.
Usage
Testing the model using transformers:
python
1from transformers import GPT2TokenizerFast, pipeline
2#for base and medium3from transformers import GPT2LMHeadModel
4#for large and mega5# pip install arabert6from arabert.aragpt2.grover.modeling_gpt2 import GPT2LMHeadModel
78from arabert.preprocess import ArabertPreprocessor
910MODEL_NAME='aubmindlab/aragpt2-base'11arabert_prep = ArabertPreprocessor(model_name=MODEL_NAME)1213text=""14text_clean = arabert_prep.preprocess(text)1516model = GPT2LMHeadModel.from_pretrained(MODEL_NAME)17tokenizer = GPT2TokenizerFast.from_pretrained(MODEL_NAME)18generation_pipeline = pipeline("text-generation",model=model,tokenizer=tokenizer)1920#feel free to try different decoding settings21generation_pipeline(text,22 pad_token_id=tokenizer.eos_token_id,23 num_beams=10,24 max_length=200,25 top_p=0.9,26 repetition_penalty =3.0,27 no_repeat_ngram_size =3)[0]['generated_text']
1python create_pretraining_data.py
2 --input_file=<RAW TEXT FILE with documents/article separated by an empty line>3 --output_file=<OUTPUT TFRecord>4 --tokenizer_dir=<Directory with the GPT2 Tokenizer files>
All models are available in the HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2 and TF1 formats.
Compute
Model
Hardware
num of examples (seq len = 1024)
Batch Size
Num of Steps
Time (in days)
AraGPT2-base
TPUv3-128
9.7M
1792
125K
1.5
AraGPT2-medium
TPUv3-8
9.7M
1152
85K
1.5
AraGPT2-large
TPUv3-128
9.7M
256
220k
3
AraGPT2-mega
TPUv3-128
9.7M
256
780K
9
Dataset
The pretraining data used for the new AraGPT2 model is also used for AraBERTv2 and AraELECTRA.
The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)
For the new dataset we added the unshuffled OSCAR corpus after we thoroughly filter it, to the dataset used in AraBERTv1 but without the websites that we previously crawled:
Assafir news articles. Huge thank you for Assafir for giving us the data
Disclaimer
The text generated by AraGPT2 is automatically generated by a neural network model trained on a large amount of texts, which does not represent the authors' or their institutes' official attitudes and preferences. The text generated by AraGPT2 should only be used for research and scientific purposes. If it infringes on your rights and interests or violates social morality, please do not propagate it.
If you used this model please cite us as :
@inproceedings{antoun-etal-2021-aragpt2,
title = "{A}ra{GPT}2: Pre-Trained Transformer for {A}rabic Language Generation",
author = "Antoun, Wissam and
Baly, Fady and
Hajj, Hazem",
booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
month = apr,
year = "2021",
address = "Kyiv, Ukraine (Virtual)",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2021.wanlp-1.21",
pages = "196--207",
}
Acknowledgments
Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the AUB MIND Lab Members for the continuous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.
