This repository contains the weights for my custom implementation of GPT-2.
For the codebase, please refer to the main repository:
GitHub LinkMy implementation has a different architecture structure than the original OpenAI release, but it aligns with the Hugging Face format available here:
Hugging Face GPT-2 (openai-community)To ensure compatibility, I’ve included both the Hugging Face-compatible weights and the weights from my own training.
To use the weights with your GPT-2 implementation, follow the example below. The model is defined here
model.py as
GPT2.
Below are two examples demonstrating how to load the model weights depending on whether you are using my GPT-2 or the Hugging Face-compatible weights.
1import torch
2from model import GPT2
3
4model_config = {
5 "n_blocks": 12,
6 "seq_len": 1024,
7 "n_embd": 768,
8 "n_head": 12,
9 "vocab_size": 50304,
10 "dropout": 0.1 # Used for training
11}
12
13model = GPT2(**model_config)
14ckpt_path = 'gpt2.pth'
15state_dict = torch.load(ckpt_path, map_location='cpu')["model_state_dict"]
16model.load_state_dict(state_dict)
17model.eval()
1
2import torch
3from model import GPT2
4
5hf_model_config = {
6 "n_blocks": 12,
7 "seq_len": 1024,
8 "n_embd": 768,
9 "n_head": 12,
10 "vocab_size": 50257,
11 "dropout": 0.0
12}
13
14model = GPT2(**hf_model_config)
15ckpt_path = 'HF_Weights.pth'
16state_dict = torch.load(ckpt_path, map_location='cpu')["model_state_dict"]
17model.load_state_dict(state_dict)
18model.eval()
19
