A faithful ~52M parameter replica of the GPT-3 architecture (Brown et al., 2020), trained from scratch on a mixture of TinyStories, FineWeb Edu, and a full scrape of the Minecraft Wiki captured on June 5, 2026 — eleven days before the Chaos Cubed drop (Java Edition 26.2) released publicly.
Model Description
GPT-3 Tiny replicates every architectural decision from the GPT-3 paper at a reduced scale:
Hyperparameter
This model
GPT-3 Small (125M)
Source
n_layers
8
12
Table 2.1
d_model
512
768
Table 2.1
n_heads
8
12
Table 2.1
d_head
64
64
d_model / n_heads
d_ff
2048
3072
4 × d_model
n_ctx
2048
2048
Section 2.1
n_vocab
50,257
50,257
GPT-2 BPE
Total params
~52M
125M
—
Architectural choices (all matching GPT-3):
Decoder-only Transformer (no cross-attention)
Pre-LayerNorm (LayerNorm before each sub-block, as in GPT-2)
Learned positional embeddings
Multi-head causal (masked) self-attention
FFN: Linear → GeLU → Linear, hidden = 4 × d_model
Tied input/output embedding weights
Final LayerNorm after last block
Training
Hardware: Google Colab T4 GPU (16 GB VRAM) Precision: fp16 mixed precision Optimizer: AdamW (β₁=0.9, β₂=0.95, ε=1e-8) Learning rate: 6×10⁻⁴ with cosine decay and linear warmup Weight decay: 0.1 (weight matrices only; biases and LayerNorm excluded) Gradient clipping: 1.0 Batch size: 32 sequences × 512 tokens Tokenizer: GPT-2 BPE via tiktoken (50,257 vocab)
Training Data
Dataset
Description
Approx. size (of slice)
TinyStories
Simple children's stories
~180 MB
FineWeb Edu
Filtered educational web text
~180 MB
Minecraft Wiki
Full scrape, June 5, 2026
~75 MB (after some cleaning)
The Minecraft Wiki scrape is the most distinctive component. It was captured eleven days before the Chaos Cubed game drop (Java Edition 26.2, released June 16, 2026), and therefore contains snapshot history pages, block/item tables, and feature descriptions for sulfur cubes, sulfur caves, cinnabar blocks, and the new 26.x versioning format — content that did not exist in any prior public model's training data.
Known data issue: No <|endoftext|> separator token was inserted between documents during packing. This causes cross-domain contamination in generation (see Limitations).
Intended Uses
Educational: Understanding GPT-3 architecture behavior at small scale
Research: Studying hallucination patterns, recency underrepresentation, and cross-domain bleed in small LMs
Minecraft community: Exploring model knowledge of pre-release Chaos Cubed content
Experimentation: Fine-tuning on domain-specific corpora
Out-of-scope uses
This model is not suitable for factual question answering, any production application, or any task requiring reliable knowledge retrieval. It hallucinates fluently and confidently.
Evaluation & Observed Behavior
The model was evaluated qualitatively through extensive prompted generation on Minecraft Wiki-style inputs. A detailed hallucination taxonomy was identified:
What the model gets right
MediaWiki template syntax:{{HistoryLine}}, {{BlockLink}}, {{ItemLink}}, [[File:...|32px]], |- table rows — all generated correctly and consistently
New versioning format: Correctly generates dev=26.2, exp=Drop 2 of 2026, dev=snap1 through dev=snap6 — the new year.drop format introduced in December 2025
Chaos Cubed content: Correctly recalls sulfur cubes, sulfur caves, cinnabar blocks, and the minecraft:sulfur_cube_archetype behavioral system (including the real archetype name regular)
Best observed output:
|{{HistoryLine|||dev=15w47a|Prior to [[Java Edition 1.13/Flattening|''The Flattening'']],
this item's numeral ID was 69.}}
Correct snapshot, correct historical event, correct wiki link format, correct numeric ID for Lever.
Hallucination taxonomy
Type
Example
Root cause
Temporal detachment
Pale Moss Carpet in 19w06a (correct block, wrong era by 5 years)
Spurious co-occurrence in training data
Stable false belief
23w10a generated repeatedly as real snapshot (doesn't exist)
Burned-in wrong association, fires reliably
Recency underrepresentation
Sulfur cube content correctly known but sometimes attributed to fake 23w10a instead of real 26.2
YYwNNa format has thousands of training pages; 26.x has very few
Correct mechanic template, sampled wrong target each time
The sulfur cube finding
Across more than eight independent generations, the model consistently produced sulfur cubes with correct metadata (dev=26.2, exp=Drop 2 of 2026) and partially correct mechanics (the regular archetype is a real entry in the minecraft:sulfur_cube_archetype registry). The model incorrectly placed sulfur caves in the Nether rather than the Overworld — likely because sulfur is Nether-associated in thousands of Minecraft mod pages that predated the official feature. This is not hallucination; it is accurate knowledge with one wrong spatial association.
The model's consistent generation of sulfur cube content before the drop's public release is a concrete example of recency underrepresentation: the model has the knowledge, but the old YYwNNa snapshot format is so statistically dominant that it sometimes overwrites the correct 26.2 attribution.
Limitations
Hallucinates confidently and fluently. All outputs should be treated as fiction unless independently verified.
No document separators in training. Cross-domain bleed (Minecraft Wiki → FineWeb Edu → TinyStories mid-generation) occurs regularly and is not fixable without retraining with <|endoftext|> separators.
52M parameters is very small. Coherent factual recall requires ~10× more parameters at minimum.
No instruction tuning or RLHF. This is a raw language model; it does not follow instructions.
Biome placement errors for new content. Sulfur caves are Overworld; model places them in the Nether.
Fake snapshot names.23w10a, 19w50a, 24w40a, and others are generated with high confidence but do not exist.
Citation
If you use this model or reference the hallucination analysis in research, please cite:
bibtex
1@misc{popa2026gpt3tiny,
2 author = {Popa, Mihai},
3 title = {GPT-3 Tiny: A 52M Parameter GPT-3 Replica Trained on the Minecraft Wiki},
4 year = {2026},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/MihaiPopa-1/GPT-3-Tiny-Exp-52M}
7}
