You can deploy this model and start earning money today!

achavan1211/
shakespeare-gpt-bpe

Name: shakespeare-gpt-bpe
Availability: InStock

Views

No views yet

Shakespeare GPT (BPE subword tokenizer)

A small decoder-only Transformer (~2.2M parameters) trained from scratch on the Tiny Shakespeare dataset. This is the BPE subword version: instead of a 65-character vocabulary, it trains a ~1000-token Byte-Pair Encoding tokenizer on the corpus and trains the transformer over those subword tokens.

Model Details

Architecture: decoder-only Transformer
Tokenizer: Byte-Pair Encoding (huggingface tokenizers)
Vocab size: 1000 (BPE subwords)
Embedding dim: 192
Number of heads: 4
Number of layers: 4
Context length (block size): 256
Total parameters: 2.21M

Training

Dataset: Tiny Shakespeare
Optimizer: AdamW (lr=0.0003, weight_decay=0.1)
Iterations: 3000 (best checkpoint @ step 2999)
Best val loss: 3.7030 (2.116 BPC)

Usage

Custom PyTorch model. See model.py for the architecture, config.json for hyperparameters, and tokenizer.json for the BPE tokenizer.

python

import json, torch
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
from model import GPTLanguageModel

repo = "achavan1211/shakespeare-gpt-bpe"

config = json.load(open(hf_hub_download(repo, "config.json")))
tokenizer = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))

encode = lambda s: tokenizer.encode(s).ids
decode = lambda l: tokenizer.decode(l)

model_cfg = {k: v for k, v in config.items() if k != "tokenizer"}
model = GPTLanguageModel(**model_cfg)
model.load_state_dict(torch.load(hf_hub_download(repo, "pytorch_model.bin"), map_location="cpu"))
model.eval()

context = torch.zeros((1, 1), dtype=torch.long)
print(decode(model.generate(context, max_new_tokens=500)[0].tolist()))