It's one of my fun project to reproduce the GPT 2 model that can make some Vietnamese poem. The model is trained on Truyen Kieu dataset, which itself is a six-eight poem (a type of poetry in Vietnamese)

Just like any model, you can download the safetensors file and then load to a torch model that is define in gpt.py and gpt_utils.py

The model can be create:
model = GPT(512, 129, 512, 32, 6)

and then you could load the model like a normal loading procedure from torch load function

The model also has a function called generate() which will generate a poem like this:

context = torch.zeros((1, 1), dtype=torch.long, device=device)
poem = model.generate(context, max_new_tokens=1000, block_size=32)

Output should look like this:
Cạnh kha trận lẹ ngồi chân ra dong.
ở gươm vòng xiêm ra đến tay,
Giập rằng, cất hôi đám dài chút thân!
Bạc bên bên vào mộng gươm lặng,
So nàng, thì vội được chay oan .
Mây trần nghe phải cỗi hoàng,

license: mit tags: