SmolLM2-135M reproduction (logit parity on CPU fp32; not bit-exact on GPU)
A single-file PyTorch reimplementation of HuggingFaceTB/SmolLM2-135M, written from the architecture up and checked against the official Hugging Face weights.
Parity, stated honestly
On CPU in fp32 it is exact. Max absolute logit delta 0.000e+00 across the full logit vector, on the 5-token prompt "The capital of France is", token ids 504, 3575, 282, 4649, 314, argmax 260 which decodes to " the", against a tolerance gate of 1e-3.
On GPU it is not exact, and I am not rounding that away. comparison_with_hf.md records max absolute delta 4.72e-05 on the final logits, 1.95e-03 per-layer at layer 14, 4.01e-05 at long context, and 0.072 against 0.080 under sampling. That per-layer 1.95e-03 exceeds the same 1e-3 gate the CPU test passes. The cause is SDPA backend dispatch: the reference passes an explicit attention mask while this implementation passes is_causal=True, and the two select different kernels. Those GPU numbers exist only as prose in a markdown file; comparison_with_hf.json was never written.
The pytest form of the check is stronger than the script form. It also asserts the parameter count at 134,515,008, checks the tied embedding by data pointer, runs a 512-token long-context case, and compares per-layer activations across all 30 blocks. But it skips rather than fails when the network or the model cache is unavailable, so a green pytest run is not by itself proof that parity was verified.
What 15.371 is, and what it is not
Perplexity 15.370989 for this implementation against 15.370990 for the reference, a delta of 8.7e-07. Both measured live in fp32 on GPU.
That is an implementation-equivalence check. The official SmolLM2-135M weights were loaded into this reimplementation's model class and scored alongside the reference implementation on the same data. It says the two forward passes agree. It is not a quality result for any checkpoint trained in this project, and it must not be attached to either local checkpoint.
The eval recipe behind it is also non-standard, so the number is not comparable to published wikitext-2 perplexities. Salesforce/wikitext, wikitext-2-raw-v1, validation split, with blank rows filtered out before joining on double newlines. A sliding window at sequence length 1024 and stride 512 with no -100 masking on the overlap, so 61 windows times 1023 gives 62,403 scored targets spread over only 31,743 distinct positions, a duplication factor of 1.97; 30,660 positions are scored twice and 1,083 once. The token count is capped at the first 32,000 of the split's 268,140 tokens, which is 11.84 percent coverage.
Architecture
RMSNorm with fp32 upcast, rotary position embeddings, grouped-query attention, and a SwiGLU MLP, in a single file with no framework abstractions. All thirteen SmolLM2Config defaults match the official config.json.
An earlier version of this card said the model was "roughly 317 lines" and the repository claimed 198 lines in model.py (that claim was removed from the repository on 2026-08-05). Neither is verifiable: there is no model.py, only model_full.py. The line count has been dropped rather than guessed.
Two sharp edges worth knowing. attention_dropout is declared in both configs but never read, because dropout_p is hardcoded to 0.0. And causal masking is selected by is_causal=(attention_mask is None), so passing an attention_mask silently disables causal masking.
Why bit-exact and not "close enough"
Language model reimplementations fail silently. Pick the wrong RoPE convention, split-halves instead of interleaved, and the weights still load, the model still runs, and the output is quietly wrong. No error, no crash. The same is true of a missing fp32 upcast in RMSNorm, the wrong GQA repeat axis, and a pre/post-norm swap. A parity gate against the official weights catches all of them at once, which is why it is the first thing built here rather than the last.
Continued pretraining on TinyStories
Metric
Before
After
TinyStories validation perplexity
6.8945
3.7900
A 45.03 percent reduction. The model also stopped collapsing into repetition loops.
Provenance, because it changes how this should be read: the run starts from the official SmolLM2-135M safetensors and continues training on TinyStories. It is a fine-tune, not a from-scratch model, which is why the frontmatter declares base_model_relation as finetune. The 6.8945 baseline is the official model's own score on this eval, not a random-initialisation floor.
The repository prose quotes 3.7893. That belongs to an earlier run with a 137.3 minute wall clock, not to this one.
Eval recipe: the first 1,040 non-empty validation stories, tokenized with add_special_tokens=False plus one EOS each, packed to 200,068 tokens, then 195 non-overlapping windows of 1024 at stride 1024, giving 199,485 target tokens. bf16 model on CUDA with logits cast to fp32 before cross-entropy. Note the stride differs from the wikitext eval above, so the two are not the same harness.
Training: a 100,000,000-token budget, 99,999,744 tokens actually seen, 24,414 steps at 4,096 tokens per step, sequence length 1024, drawn from 102,000,116 packed training tokens across 99,609 windows. bf16. Warmup-stable-decay with a 3e-4 peak, 200 warmup steps and linear decay to zero over the final 20 percent. Wall clock 116.1 minutes at 14,356 tokens per second as a cumulative average.
Loss: best single-step 0.9088 at step 22,353. First bucket mean 1.586, final bucket mean 1.3138. An earlier "1.316 (last)" figure was off by one bucket.
Gaps in this result that I am not papering over
The optimizer hyperparameters for this run were never recorded. There is no args line, no training_recipe key and no grad-norm column in the log. The AdamW betas of 0.9 and 0.95, eps 1e-8, weight decay 0.01, gradient clipping 1.0 and seed 0 that appear elsewhere are a later script's argparse defaults, not a record of what ran. Worse, training_recipe_resolved.json holds values copied from the upstream nanotron config_smollm2_135M.yaml, learning rate 0.003, warmup 2000, sequence length 2048 and two million steps, which describe the original SmolLM2 pretraining run and not this one. Micro-batch 4 with gradient accumulation 1 is prose only; the log records just the product, 4,096.
The training script on disk is not the version that produced this run. Six independent lines of evidence show drift, and the exact source that ran is not recoverable.
No out-of-distribution, catastrophic-forgetting or downstream measurement exists for the TinyStories checkpoint. eval_after_vs_base.py never ran, scripts/run_lm_eval.sh never ran, and results/lm_eval is absent. So the 45 percent gain on TinyStories is a gain on TinyStories, with nothing measuring what it cost elsewhere.
Hardware: an NVIDIA GB10 is named in prose, but the training log records only "Device: cuda".
Checkpoints
checkpoint_tinystories.pt, 269,144,681 bytes, zip with method store, keys model, config, step, tok_seen, baseline_ppl and trained_ppl, 273 tensors all in bf16. It is not resumable: no optimizer, scheduler or RNG state is saved. It is gitignored and untracked. Its sha256 is 78e82767ea1bde589f97b915fc2133e0a6040993f859846a2e4a1fe7f2bb39b0. It is no longer a single copy: on 2026-08-05 the raw file was archived to a private Hugging Face repository, and the remote copy's sha256 was read back from the Hub and matched the local digest exactly. The project ledger still records no checksum for it, because the TinyStories run predates ledger adoption and has no run entry at all.
The embedded config reads vocab_size 49,152, hidden_size 576, intermediate_size 1,536, 30 layers, 9 attention heads, 3 key-value heads, max_position_embeddings 8,192, rope_theta 1e5, rms_norm_eps 1e-5, tied word embeddings, no attention bias. 162,826,560 elements including the tied lm_head duplicate, 134,515,008 unique. The state-dict keys are already in Hugging Face Llama form, so conversion needs no rename map.
There is also a checkpoint.pt of 538,173,921 bytes. It is a 150-step random-initialisation toy demo trained on wikitext-2 train inside a notebook cell, and it is not SmolLM2 weights. It is not publishable as such and is not published. The cell's own comment says wikitext-103 while the code loads wikitext-2; the comment is stale.
The converter now runs. scripts/export_to_hf.py takes a checkpoint and an output directory, pulls the base config and tokenizer from HuggingFaceTB/SmolLM2-135M, and writes config.json at 793 bytes, generation_config.json at 194 bytes, model.safetensors at 269,060,552 bytes, tokenizer_config.json at 764 bytes, tokenizer.json at 3,522,871 bytes and training_recipe.json at 204 bytes. The result round-trips: it loads through stock AutoModelForCausalLM with no trust_remote_code, reports 134,515,008 parameters, and generates coherent TinyStories prose.
The fp32 widening is fixed. The script previously had no dtype cast, so the checkpoint's bf16 tensors were widened when they loaded into a default-dtype module and 269 MB became 538 MB. It now takes --dtype, defaulting to bf16, which matches both the training dtype and the base model's own published torch_dtype; --dtype fp32 restores the old behaviour. The bf16 export is lossless: all 272 stored tensors compare bit-identical to the source checkpoint under torch.equal, worst absolute difference 0.0.
One thing remains, inherited unchanged from the base model: bos_token_id and eos_token_id are both 0 and pad_token_id is null, so generation emits a pad-token warning unless you pass pad_token_id. No special_tokens_map.json is written, but that is a transformers v5 change rather than a defect -- bos, eos and unk are all present in tokenizer_config.json.
The corrections described on this card are on the default branch. PR #5 merged on 2026-08-05 and main is now at 6911743, which includes the 1.3138 bucket-mean fix and the removal of the line count for a model.py that does not exist. The repository and this page agree.
Reproduce
Parity:
cd "SmolLM2-134(base)" && python verify.py
pytest tests/ -v
A wording trap in that first command: verify.py's docstring mentions a bf16 tolerance, but the code and the recorded run are fp32. The docstring is stale, not the result.
The 15.371 perplexity has no standalone script. It lives in cell index 14 of results.ipynb, regenerated with:
Three hazards before anyone runs that. It overwrites results.ipynb in place, wiping the existing outputs. It re-executes a 150-step training cell that overwrites ../checkpoint.pt. And there is no CUDA-availability guard, so it will not fail gracefully on a CPU-only machine.
Environment actually used: Python 3.12.11, torch 2.11.0+cu130, CUDA 13.0, cuDNN 91900, driver 580.142, NVIDIA GB10. The cuDNN and driver versions are not recorded anywhere in the repository. transformers 5.8.0 and datasets 4.8.5 are pinned in pyproject.toml but were not stamped by the run, and requirements.txt contradicts pyproject with torch>=2.4 and transformers>=4.40. Dataset and model revisions were resolved from cache rather than pinned in code: SmolLM2-135M at 93efa2f0 and TinyStories at f54c09fd.
Limitations
A faithful architecture reproduction with a working train and eval harness. It is not a from-scratch retrain to the reference model's quality. Parity holds on CPU in fp32 and does not hold to the same tolerance on GPU. Single machine, single GPU. No KV cache at the time of writing. Both eval harnesses here are non-standard and their numbers should not be compared against published leaderboard figures.
