A decoder-only transformer built entirely from scratch — every forward and backward pass
derived and coded by hand, with no autograd, no PyTorch, no HuggingFace transformers.
Implemented twice: once in NumPy, once in CUDA/C++ (this checkpoint is from the CUDA trainer).
Architecture
A LLaMA-style pre-norm decoder stack:
82.9M parameters
d_model = 512
12 layers, 8 attention heads (d_head = 64)
SwiGLU feed-forward (d_ff = 2026), with a learned Swish-gate β
RMSNorm instead of LayerNorm
Learned positional embeddings (no RoPE)
Context window: 256 tokens, no KV cache — every generated token is a full forward
pass over the whole context
Custom byte-level BPE tokenizer, also trained from scratch in C++
Corpus: ~4.01B tokens (~10GB text)
Optimizer: AdamW with linear warmup + cosine LR decay, global gradient-norm clipping,
label smoothing 0.05, dropout 0.1
Random-window sampling: each step draws a uniformly random contiguous 256-token window
from anywhere in the corpus (not sequential epochs)
This checkpoint: step 698,000 (training run complete), loss ~2.7, perplexity ~15
This is a base language model — pretrained on raw text only, not instruction-tuned or
RLHF'd. It completes text fluently but has no grounded factual knowledge or chat behavior;
treat its output as free-associative continuation, not assistant responses.
Checkpoint format
latest.ckpt is the CUDA trainer's native binary format (TFCKPT1 magic header), not a
PyTorch state_dict. It carries its own architecture header (step, vocab size, d_model,
heads, layers, d_ff, max_len) so it's self-describing. See the project repo for the
loader (utils/ckpt_convert.py, cuda/include/checkpoint.cuh) and inference code
(generate.py, serve.py).
Repo contents
latest.ckpt — the checkpoint itself.
tokenizer/tokenizer.bbpe (+ merges.txt, vocab.json) — the from-scratch byte-level
BPE tokenizer this checkpoint was trained with. Must match exactly; a different
tokenizer/vocab size will silently produce garbage.
The training corpus itself isn't included here (too large / not redistributable); bring
your own plain-text corpus to resume training.
Then resume training against your own corpus (see the project repo's
docs/continue-training-from-usb.md for a full walkthrough of resuming from a checkpoint):
