CreekwardGoat-500K is a compact, decoder-only transformer developed to explore the capabilities of tiny language models. Despite its remarkably small scale of around 500 thousand parameters, it achieves very competitive performance within its size class on established evaluation benchmarks, including BLiMP and ARC-Easy.
This model is fully transparent in training and reproducible from scratch.
Download the training data:
python get_training_data.py
Start training:
python train.py
Training configurations including seed and number of training tokens can be modified in config.py.
If you leave the files unchanged, theoretically, this should reproduce the training trajectory as well as the final model output.
Training Data
The model was trained on a 300 million token subset of FineWeb-Edu-Dedup, taken from the HuggingFaceTB/smollm-corpus repository. The specific shard was selected due to its relatively small size to reduce storage and preprocessing overhead.
This model is an experimental personal research artifact and is not intended for production or consumer use. It is released as a base (pre-trained) model and does not support conversational interaction. Due to its extremely small scale (≈500K parameters), this model is fundamentally incapable of reliable output. It may generate:
Factually inaccurate or hallucinated content
Grammatically garbled or nonsensical text
Unexpected or biased outputs
Generated outputs do not represent the author's views. Users are solely responsible for verifying outputs, ensuring compliance with all applicable laws, and bearing all consequences of use or redistribution. The author accepts no responsibility for model outputs or downstream use.
