A 115.1M-parameter decoder-only transformer trained from scratch on TinyStories for 16M tokens, using the v11 tokenizer.

This is the base pretrain only: phase 1, no frozen-FFN attention retrain, and no maths mid-training. It writes competent children's-story English and it cannot do arithmetic — it narrates straight past the place a number belongs rather than putting a wrong number there. That is the intended state of this checkpoint, not a defect: it is the starting point for the mid-training experiments in tinystories-train-video.

Not an AutoModel — TinyModel is a 3-file Gemma-shaped decoder (RMSNorm, RoPE, GQA, gated FFN, tied embeddings), shipped in this repo under tiny_model_v11/.

That exact round-trip — download, build, load, generate — is run against the uploaded files at publish time. It is not an untested snippet.

The embedding table is 36.5M parameters — 32% of the whole model is its vocabulary lookup, which is what a 71k vocabulary costs at this width.

rope_freqs is a complex64 buffer in the state dict. It is derived from dim/n_heads/max_seq and recomputed on construction, so it round-trips but carries no learned information.

Identity is the content hash, not the Hub revision — re-pushing identical bytes mints a new commit oid, and a README edit does too. Join on these:

The tokenizer sha is the same value this checkpoint's training run wrote into its meta.json as tokenizer_hash, and the same one chrishayuk/v11-tokenizer publishes. A checkpoint driven by a different tokenizer produces fluent nonsense rather than an error, so that join is checked mechanically at publish time, not asserted here.

The dataset revision is pinned, so the document set is reproducible and held-out text can be shown never to have been trained on.

emergence.json carries the generations captured at each milestone, plus the loss trace. Greedy, 30 new tokens, same prompts throughout:

Only the final row's weights are published here; the earlier rows are the same run mid-flight.

It cannot do arithmetic. Nothing in TinyStories teaches addition, and number words in that corpus are narrative texture rather than quantities — "once upon a time there were two" is an idiom the model learns the way it learns "happily ever after".

The interesting part is how it fails. It does not answer with a wrong number; it carries on telling the story, straight past the place a number belongs.

Its own generation at the final milestone, greedy, from this run:

It has also seen 16M tokens, which is not converged. Continued training on more or less anything improves it, so do not read an improvement after mid-training as evidence that the mid-training data helped specifically.

Research and teaching on compact language models: mid-training, tool-use / tool-call training, and measuring the difference between memorising a distribution and learning an algorithm. Not intended for any production use, and it has no safety training of any kind.

provenance.json records the model and tokenizer hashes, the corpus pin, and the run config. The chuk-datasets catalog holds a content-addressed, bit-reproducible tokenization of the same corpus (tiny-model/v11-rust-tokenized-phase1); these weights came from the pinned HF revision instead, which is reproducible given the seed but not bit-identical. That distinction is recorded rather than glossed.