A GPT-2
(Generative Pretrained Transformer-2) model is a transformer based architecture for Causal Language Modeling, meaning it's required a left token/word as an input prompt
for generating the right/next token, developed by Open AI
{Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya}.
See the paper here:
https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdfSince GPT-2 is an unsupervised model and trained using an unlabelled of text sequences without any explicit supervision,
the clarity and output of this model often comes with randomness. To overcome this issue we have to create a specific seed for determined output.
Supported language for this model is only English (get from GPT-2 pretrained model) and Indonesian (fine tune using Indonesian Wikipedia Dataset).
1>>> from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM, set_seed
2
3>>> model_name = 'anugrahap/gpt2-indo-textgen'
4>>> tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side='left')
5>>> model = AutoModelForCausalLM.from_pretrained(model_name, pad_token_id=tokenizer.eos_token_id)
6>>> generator = pipeline('text-generation', model=model, tokenizer=tokenizer)
7
8>>> #set_seed(1)
9>>> result = generator("Skripsi merupakan tugas akhir mahasiswa", min_length=10, max_length=30, num_return_sequences=1)
10>>> result[0]["generated_text"]
