Views
No views yet
gpt2 folder and can trains models from the minimaxir/gpt-2-simple repository.
These models were trained using the lamb optimizer and follow the same architecture as gpt2 and are fully compatible with the transformers library.grover architecture. You can use the pytorch classes found in grover/modeling_gpt2.py as a direct replacement for classes in the transformers library (it should support version v4.x from transformers).
Both models are trained using the adafactor optimizer, since the adam and lamb optimizer use too much memory causing the model to not even fit 1 batch on a TPU core.transformers:1from transformers import GPT2TokenizerFast, pipeline
2#for base and medium
3from transformers import GPT2LMHeadModel
4#for large and mega
5# pip install arabert
6from arabert.aragpt2.grover.modeling_gpt2 import GPT2LMHeadModel
7
8from arabert.preprocess import ArabertPreprocessor
9
10MODEL_NAME='aubmindlab/aragpt2-medium'
11arabert_prep = ArabertPreprocessor(model_name=MODEL_NAME)
12
13text=""
14text_clean = arabert_prep.preprocess(text)
15
16model = GPT2LMHeadModel.from_pretrained(MODEL_NAME)
17tokenizer = GPT2TokenizerFast.from_pretrained(MODEL_NAME)
18generation_pipeline = pipeline("text-generation",model=model,tokenizer=tokenizer)
19
20#feel free to try different decoding settings
21generation_pipeline(text,
22 pad_token_id=tokenizer.eos_token_id,
23 num_beams=10,
24 max_length=200,
25 top_p=0.9,
26 repetition_penalty = 3.0,
27 no_repeat_ngram_size = 3)[0]['generated_text']transformers:1python create_pretraining_data.py
2 --input_file=<RAW TEXT FILE with documents/article separated by an empty line>
3 --output_file=<OUTPUT TFRecord>
4 --tokenizer_dir=<Directory with the GPT2 Tokenizer files>python3 run_pretraining.py \\\n --input_file="gs://<GS_BUCKET>/pretraining_data/*" \\\n --output_dir="gs://<GS_BUCKET>/pretraining_model/" \\\n --config_file="config/small_hparams.json" \\\n --batch_size=128 \\\n --eval_batch_size=8 \\\n --num_train_steps= \\\n --num_warmup_steps= \\\n --learning_rate= \\\n --save_checkpoints_steps= \\\n --max_seq_length=1024 \\\n --max_eval_steps= \\\n --optimizer="lamb" \\\n --iterations_per_loop=5000 \\\n --keep_checkpoint_max=10 \\\n --use_tpu=True \\\n --tpu_name=<TPU NAME> \\\n --do_train=True \\\n --do_eval=False| Model | Optimizer | Context size | Embedding Size | Num of heads | Num of layers | Model Size / Num of Params |
|---|---|---|---|---|---|---|
| AraGPT2-base | lamb | 1024 | 768 | 12 | 12 | 527MB / 135M |
| AraGPT2-medium | lamb | 1024 | 1024 | 16 | 24 | 1.38G/370M |
| AraGPT2-large | adafactor | 1024 | 1280 | 20 | 36 | 2.98GB/792M |
| AraGPT2-mega | adafactor | 1024 | 1536 | 25 | 48 | 5.5GB/1.46B |
HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2 and TF1 formats.| Model | Hardware | num of examples (seq len = 1024) | Batch Size | Num of Steps | Time (in days) |
|---|---|---|---|---|---|
| AraGPT2-base | TPUv3-128 | 9.7M | 1792 | 125K | 1.5 |
| AraGPT2-medium | TPUv3-8 | 9.7M | 80 | 1M | 15 |
| AraGPT2-large | TPUv3-128 | 9.7M | 256 | 220k | 3 |
| AraGPT2-mega | TPUv3-128 | 9.7M | 256 | 780K | 9 |
@inproceedings{antoun-etal-2021-aragpt2,
title = "{A}ra{GPT}2: Pre-Trained Transformer for {A}rabic Language Generation",
author = "Antoun, Wissam and
Baly, Fady and
Hajj, Hazem",
booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
month = apr,
year = "2021",
address = "Kyiv, Ukraine (Virtual)",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2021.wanlp-1.21",
pages = "196--207",
}