A 109-million-parameter causal language model trained from scratch on open-webtext using only free compute (Google Colab + Kaggle).
The project is a tribute to Tom Platz’s “no-excuses” mindset: we are not prisoners of our hardware (or genetics)—we squeeze every rep out of what we have. Owing to hardware limitations, the model was trained only on a fraction of OpenWebText (around 513M tokens), but the architecture allowed it to gain generative capabilites and grammatically coherent sentences, while being able to maintain some context. This was undertaken as an educational project demonstrating proof of concept.