MiniGPT: A GPT Model Fine-Tuned on Shakespeare Texts
This is a custom GPT model fine-tuned on a small subset of Shakespeare's works. The model is built using PyTorch and Transformers, and is designed to generate creative text in the style of Shakespeare.
🚀 Overview
This project demonstrates the process of fine-tuning a small version of GPT on a custom dataset and deploying it to Hugging Face for easy use and access. The model has been trained using a tiny Shakespeare dataset (tiny_shakespeare.txt), and is capable of generating creative text based on a given prompt.
🧑💻 Setup
To use or replicate this model, follow the steps below to get started.
1. Installation
First, install the required dependencies:
📚 Dataset
This model was fine-tuned on a small dataset of Shakespeare's works. You can use your own dataset by following the structure below
Load and tokenize data
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token # Important for padding
Load your dataset
file_path = "path/to/your_dataset.txt"
with open(file_path, "r", encoding="utf-8") as f:
raw_text = f.read()
encoded = tokenizer(raw_text, return_tensors="pt", truncation=True, padding=True, max_length=512)
input_ids = encoded["input_ids"]
🧑💻 Training
To fine-tune the model, follow the instructions below:
from transformers import TrainingArguments
import wandb
wandb.login()
Set up W&B logging configuration
wandb.init(project="mini-gpt", config={
"learning_rate": 0.001,
"epochs": 5,
"batch_size": 16
})
Training loop
train(model, optimizer, epochs=5, encoded_dataset=encoded_dataset, batch_size=4)
💾 Saving the Model
Once the model is trained, you can save it like this:
torch.save(model.state_dict(), "my-mini-gpt-model/pytorch_model.bin")
tokenizer.save_pretrained("my-mini-gpt-tokenizer")
🚀 Uploading the Model to Hugging Face
You can upload the model to Hugging Face Hub using the following commands:
from huggingface_hub import login, create_repo, upload_folder
login(token="your_huggingface_token")
repo_id = "your_username/my-mini-gpt"
create_repo(repo_id, exist_ok=True)
upload_folder(folder_path="my-mini-gpt-model", repo_id=repo_id)
upload_folder(folder_path="my-mini-gpt-tokenizer", repo_id=repo_id)
🎮 Generating Text
Once the model is uploaded to Hugging Face, you can easily generate text using the following code:
from transformers import AutoModelForCausalLM, AutoTokenizer
Load model and tokenizer from Hugging Face
model = AutoModelForCausalLM.from_pretrained("your_username/my-mini-gpt")
tokenizer = AutoTokenizer.from_pretrained("your_username/my-mini-gpt")
Generate text based on a prompt
prompt = "In the future, AI will"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_length=50,
temperature=1.0,
top_k=50,
top_p=0.95,
do_sample=True
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
📄 License
This model is licensed under the MIT License.
📝 Citation
If you use this model or code in your research, please cite it as:
Tkachenko, A. (2025). MiniGPT: Fine-tuned GPT model for text generation. Hugging Face. Available at:
https://huggingface.co/altkachenko11/my-mini-gpt
