A small decoder-only transformer (GPT-style), trained from scratch at the character level on the Tiny Shakespeare dataset. It was built to make the mechanics of self-attention, causal masking and positional embeddings visible, by implementing them by hand instead of using torch.nn.Transformer.

This is a small educational model: 818k parameters, trained for 2000 steps on CPU. It is not meant to generate usable text. See Limitations below.

Model details

Architecture: decoder-only transformer, GPT-style, implemented from scratch (custom multi-head self-attention, no torch.nn.Transformer)
Layers: 4
Attention heads: 4
Embedding size: 128
Context length (block size): 64 characters
Feed-forward hidden size: 512 (4x embedding size), GELU activation
Normalization: pre-LayerNorm around each sub-block, residual connections
Positional encoding: learned absolute position embeddings
Vocabulary: 65 characters, a character-level tokenizer built from the training corpus itself
Parameters: 818,176 trainable. The saved weights also include 4 fixed causal-mask buffers (one per layer), which are not learned.
Precision: float32
Framework: PyTorch 2.x

The model code (the GPT class and the tokenizer) is not duplicated in this repository. It lives in the source repository: https://github.com/delcenjo/transformer-from-scratch (model.py, tokenizer.py). See Usage for how to load weights from here together with that code.

Training data

Tiny Shakespeare: about 1.1 MB of text concatenated from Shakespeare's plays, originally collected by Andrej Karpathy for the char-rnn project. Source file used here: https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

1,115,394 characters, 65 distinct characters (upper and lower case letters, punctuation, whitespace, and the digit 3). Split 90/10 into train and validation; batches are sampled randomly from each split.

Training procedure

Tokenizer: character-level, vocabulary is the sorted set of distinct characters in the corpus
Optimizer: AdamW, learning rate 3e-4
Steps: 2000
Batch size: 32 sequences of 64 characters
Dropout: 0.1
Seed: 1337
Hardware: CPU (the model and context are small enough that a GPU is not needed)

Command used to train, from the source repository:

python -m chartransformer.train

which trains with exactly the defaults above (character tokenizer, 4 layers, 4 heads, 128-dim embeddings, block size 64) and writes checkpoints/model.pt.

Evaluation

Metric: validation perplexity, exp(mean cross-entropy over 200 random batches from the held-out 10% split). Perplexity is only comparable across models that share the same tokenizer (see the byte-pair row below).

Reproduced on 2026-07-12, from the trained checkpoint, with:

python -m chartransformer.evaluate --name model

Result: val perplexity 6.90 (three runs: 6.90, 6.91, 6.93). The number recorded at training time, with the same checkpoint and method, was 6.89 - consistent within normal sampling noise, since evaluate.py draws a different random subset of validation batches on every run and no evaluation seed is fixed.

The same command was also run against the other checkpoints from the source repo's ablation study, to double check the numbers in its README:

variant	val loss	perplexity (reproduced here)	perplexity (recorded at training time)
character baseline (this model)	1.93	6.90	6.89
no positional embeddings	2.08	8.06	8.04
single attention head	1.91	6.77	6.76
byte-pair tokenizer, vocab 512	3.63	37.44	37.65

Removing positional embeddings makes perplexity noticeably worse (attention alone cannot tell tokens apart by order). Going from 4 heads to 1 head barely changes it at this model size. The byte-pair row is not directly comparable to the character rows: a byte-pair token covers roughly 2 characters on average, so its perplexity is measured over a different unit.

Usage

usage.py in this repository is a complete working example: it clones (or reuses a local copy of) the source GitHub repository for the model code, downloads config.json and model.safetensors from here, and runs generation.

bash

pip install torch safetensors huggingface_hub
python usage.py --tokens 100

Minimal inline version:

python

import json, sys
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

# model code from https://github.com/delcenjo/transformer-from-scratch (clone it first)
sys.path.insert(0, "transformer-from-scratch/src")
from chartransformer.model import GPT, GPTConfig
from chartransformer.tokenizer import tokenizer_from_config

repo_id = "delcenjo/char-transformer-shakespeare"
config = json.load(open(hf_hub_download(repo_id, "config.json")))
weights = load_file(hf_hub_download(repo_id, "model.safetensors"))

model = GPT(GPTConfig(
    vocab_size=config["vocab_size"], block_size=config["block_size"],
    n_layer=config["n_layer"], n_head=config["n_head"], n_embd=config["n_embd"],
    dropout=0.0, use_position=config["use_position"],
))
model.load_state_dict(weights)
model.eval()

tokenizer = tokenizer_from_config(config["tokenizer"])
idx = torch.tensor([tokenizer.encode("\n")], dtype=torch.long)
out = model.generate(idx, max_new_tokens=100)[0]
print(tokenizer.decode(out.tolist()))

The same attention mechanism also runs live in the browser, no download needed, in two small demos: beam search and attention.

Why there is no .pt file here

The original checkpoint (checkpoints/model.pt in the source repository) is a pickle file: loading a pickle with torch.load can execute arbitrary code, which is a known issue for sharing weights on the Hub. This repository ships model.safetensors instead, a plain tensor container with no code execution, converted directly from the original state dict (checked tensor by tensor against the original after the round trip, before uploading). config.json documents the architecture and tokenizer needed to rebuild the model with the GPT class from the source repository.

Limitations

Character-level model, 818k parameters, trained for 2000 steps on CPU. It does not produce real English. At best it produces pseudo-Elizabethan text: capitalized character names followed by a colon, line breaks, punctuation, and a number of correctly spelled short words, without coherent grammar or meaning past a few words.
Context is only 64 characters, so the model has no notion of anything outside a very short window.
Built to make self-attention and positional embeddings visible and testable, not to be used as a text generator. No instruction-following, no chat behavior, no safety filtering; it only continues characters.
Evaluated only on a held-out split of the same corpus it was trained on (in-distribution). No external benchmark, no comparison against other language models.

Bias and content notes

The training data is Shakespeare's plays, so the vocabulary and style reflect Early Modern English: archaic spelling and period-specific words, and the themes present in the plays themselves. The dataset contains no personal or user data; it is public-domain literary text. Given the model's size and the fact that it only reproduces character-level patterns from this single corpus, its output is not a source of factual claims about anything and should be read as generated text, not information.

Intended use

Educational: seeing a from-scratch transformer implementation train, and inspecting what a small character-level model does and does not learn (see the ablation table above, and the source repository's tests). Not intended for production text generation or as a base for fine-tuning toward real use cases.

License

MIT, same license as the source repository: https://github.com/delcenjo/transformer-from-scratch/blob/main/LICENSE

Citation

Training data is Tiny Shakespeare, collected by Andrej Karpathy for the char-rnn project: https://github.com/karpathy/char-rnn