Views
No views yet
gpt2 folder and can trains models from the minimaxir/gpt-2-simple repository.
These models were trained using the lamb optimizer and follow the same architecture as gpt2 and are fully compatible with the transformers library.grover architecture. You can use the pytorch classes found in grover/modeling_gpt2.py as a direct replacement for classes in the transformers library (it should support version v4.x from transformers).
Both models are trained using the adafactor optimizer, since the adam and lamb optimizer use too much memory causing the model to not even fit 1 batch on a TPU core.arabert library.transformers:trust_remote_code flag, and can be used as follows:1from transformers import AutoModelForCausalLM, pipeline
2
3from arabert.preprocess import ArabertPreprocessor
4
5MODEL_NAME='aubmindlab/aragpt2-large'
6arabert_prep = ArabertPreprocessor(model_name=MODEL_NAME)
7
8text=""
9text_clean = arabert_prep.preprocess(text)
10
11model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, trust_remote_code=True)
12tokenizer = GPT2TokenizerFast.from_pretrained(MODEL_NAME)
13generation_pipeline = pipeline(
14 "text-generation", model=MODEL_NAME, trust_remote_code=True
15)
16
17#feel free to try different decoding settings
18generation_pipeline(text,
19 pad_token_id=pipeline.tokenizer.eos_token_id,
20 num_beams=10,
21 max_length=200,
22 top_p=0.9,
23 repetition_penalty = 3.0,
24 no_repeat_ngram_size = 3)[0]['generated_text']
25>>>transformers:1python create_pretraining_data.py
2 --input_file=<RAW TEXT FILE with documents/article separated by an empty line>
3 --output_file=<OUTPUT TFRecord>
4 --tokenizer_dir=<Directory with the GPT2 Tokenizer files>1python3 run_pretraining.py \
2 --input_file="gs://<GS_BUCKET>/pretraining_data/*" \
3 --output_dir="gs://<GS_BUCKET>/pretraining_model/" \
4 --config_file="config/small_hparams.json" \
5 --batch_size=128 \
6 --eval_batch_size=8 \
7 --num_train_steps= \
8 --num_warmup_steps= \
9 --learning_rate= \
10 --save_checkpoints_steps= \
11 --max_seq_length=1024 \
12 --max_eval_steps= \
13 --optimizer="lamb" \
14 --iterations_per_loop=5000 \
15 --keep_checkpoint_max=10 \
16 --use_tpu=True \
17 --tpu_name=<TPU NAME> \
18 --do_train=True \
19 --do_eval=False| Model | Optimizer | Context size | Embedding Size | Num of heads | Num of layers | Model Size / Num of Params |
|---|---|---|---|---|---|---|
| AraGPT2-base | lamb | 1024 | 768 | 12 | 12 | 527MB/135M |
| AraGPT2-medium | lamb | 1024 | 1024 | 16 | 24 | 1.38G/370M |
| AraGPT2-large | adafactor | 1024 | 1280 | 20 | 36 | 2.98GB/792M |
| AraGPT2-mega | adafactor | 1024 | 1536 | 25 | 48 | 5.5GB/1.46B |
HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2 and TF1 formats.| Model | Hardware | num of examples (seq len = 1024) | Batch Size | Num of Steps | Time (in days) |
|---|---|---|---|---|---|
| AraGPT2-base | TPUv3-128 | 9.7M | 1792 | 125K | 1.5 |
| AraGPT2-medium | TPUv3-8 | 9.7M | 1152 | 85K | 1.5 |
| AraGPT2-large | TPUv3-128 | 9.7M | 256 | 220k | 3 |
| AraGPT2-mega | TPUv3-128 | 9.7M | 256 | 780K | 9 |
@inproceedings{antoun-etal-2021-aragpt2,
title = "{A}ra{GPT}2: Pre-Trained Transformer for {A}rabic Language Generation",
author = "Antoun, Wissam and
Baly, Fady and
Hajj, Hazem",
booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
month = apr,
year = "2021",
address = "Kyiv, Ukraine (Virtual)",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2021.wanlp-1.21",
pages = "196--207",
}