Pre-train: Culturax Vietnamese Dataset(450GB) + AI-Hub Vietnamese Dataset(1.3GB) + Crawled Vietnamese Wikipedia Dataset(630MB) + viwik18 Dataset(1.27GB)

Training Hardware

Trained on A100 40GB GPU and 48 core CPU. Took about 17 hours to reach 80,000 steps.

Hyperparameters

Hyperparameter	Value
n_parameters	2670182400
n_layers	32
d_model	2560
n_heads	32
d_head	128
n_vocab	60000
Sequence Length	2048
Learning Rate	0.00016
Positional Encoding	Rotary Position Embedding (RoPE)

How to use

The model can be loaded using the AutoModelForCausalLM functionality:

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-pretrained")
model = AutoModelForCausalLM.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-pretrained")