You can deploy this model and start earning money today!

NataliiaM15/
decoder-shakespeare-gpt

Name: decoder-shakespeare-gpt
Availability: InStock

karpathy/tiny_shakespeare

mit

Views

No views yet

Decoder-Only Shakespeare GPT

This is a lightweight GPT-style decoder-only transformer model trained on the Tiny Shakespeare dataset (karpathy/tiny_shakespeare). It uses a custom implementation in PyTorch and supports character-level text generation.

Model Details

Architecture: Decoder-only Transformer
Layers: 2
Embedding Size: 128
Heads: 4
Sequence Length: 64
Training Epochs: 4
Tokenizer: GPT-2 tokenizer (character-level)

Training

Trained on the full Tiny Shakespeare dataset for 4 epochs using Adam optimizer and cross-entropy loss. Validation loss is tracked and logged using Weights & Biases (wandb).

Usage

python

from transformers import AutoTokenizer
import torch
from model import DecoderOnlyTransformer  # custom model class

tokenizer = AutoTokenizer.from_pretrained("NataliiaM15/decoder-shakespeare-gpt")
model = DecoderOnlyTransformer(
    vocab_size=tokenizer.vocab_size,
    embed_dim=128,
    num_heads=4,
    num_layers=2,
    seq_len=64
)
model.load_state_dict(torch.load("pytorch_model.bin"))
model.eval()

# Generate text
prompt = "ROMEO:"
input_ids = tokenizer.encode(prompt, return_tensors="pt")
# generation loop would go here...