DGPT v1-base is a 13,049,856-parameter decoder-only Transformer trained from scratch (manual forward pass, manual backward pass, manual AdamW — no autograd, no PyTorch/JAX/TensorFlow) on the TinyStories dataset. It generates short, simple, TinyStories-style children's narratives and nothing more.

Do not expect: instruction following, multi-turn conversation, reasoning, factual world knowledge, or ChatGPT-comparable capability of any kind. This model was never trained or tuned for any of those.

Model description

Model type: decoder-only Transformer, Pre-LN, GELU (tanh approx), tied token embedding / LM head (no output bias), learned positional embeddings.
Parameters: 13,049,856
Context length: 256 tokens
Vocabulary: 6,000 (locked byte-level BPE, bpe_6000.json)
Framework: none — hand-implemented NumPy/CuPy. Every layer's backward pass was independently verified against finite-difference gradient checks before training.

Architecture

param	value
vocab_size	6000
block_size	256
d_model	384
n_layer	6
n_head	6
head_dim	64
d_ff	1536
activation	GELU (tanh approx)
norm	Pre-LN
positions	learned
lm_head	tied to token embedding, no bias

Training data

TinyStories (TinyStoriesV2-GPT4-train.txt): 2,717,495 synthetically generated (GPT-3.5/ GPT-4) short stories using a deliberately small vocabulary (Eldan & Li, 2023), tokenized to 371,525,259 tokens. Licensed by its authors under CDLA-Sharing-1.0 — this model card does not redistribute the dataset itself.

Training procedure

Optimizer: manually implemented AdamW (lr, betas, weight decay applied only to matrix params — biases/LayerNorm params excluded from decay).
Stage 2 (validation run): 50k-story subset, 2000 steps, batch size 64, LR 3e-4 with warmup, used to gate correctness before full training.
Full run: resumed from the Stage 2 checkpoint, continued on the full 371.5M-token corpus at LR 3e-5, to a final checkpoint at step 5000.
Tokenizer: locked, pre-trained externally, never retrained during model training.

Hardware

1x NVIDIA Tesla T4 (Turing, SM75, 16 GB VRAM), Kaggle.
CuPy 14.0.1 as the GPU numerical execution backend (no autograd usage).
Measured throughput: ~3,300–3,440 tokens/sec at batch size 64 (directly measured, not extrapolated).

Intended use

Educational reference for from-scratch Transformer implementation (manual forward/backward/AdamW) at small scale.
Generating short, TinyStories-style children's narratives from a prompt.
Portfolio / ML-engineering demonstration.

Out-of-scope use

Any production or consumer-facing assistant use case.
Instruction following, chat, question answering, factual retrieval, reasoning tasks, or code generation.
Anything requiring broad world knowledge — the model's effective knowledge is bounded by TinyStories' simplified vocabulary and narrative style.
Any use that assumes safety alignment or content filtering — none was performed.

Evaluation

From the training notebook (full-data run, step 5000):

Train loss ≈ 3.3
Val loss ≈ 3.3–3.4
Val perplexity ≈ 27–29

No held-out benchmark suite (e.g. downstream NLP tasks) was run — TinyStories train/val loss and perplexity are the only reported metrics. Treat any numbers as approximate; see the training notebook's step-by-step log for the exact source values.

Known generation issues

Occasional run-on or abruptly concatenated sentences (short stories sometimes blend into the next without a clean boundary).
Repetition of simple phrases/character names across generations.
No factual grounding — names, objects, and events are generated freely and are not to be treated as accurate about anything.
Context is capped at 256 tokens; longer prompts are truncated from the left before generation.

How to use

python

from src.generate import load_dgpt, generate_text
from src.tokenizer import BPETokenizer

tok = BPETokenizer("tokenizer/bpe_6000.json")
model, _ = load_dgpt("model.npz")

print(generate_text(model, tok, "Once upon a time", max_new_tokens=150))

Licensing

License is marked unknown above deliberately. See this repository's main README.md → "Licensing" for the full breakdown across code, weights, tokenizer, and the TinyStories dataset — the weights and tokenizer do not have an established license and none is invented here.

dipan004/DGPT

DGPT v1-base