A 1,002,720-parameter decoder-only base language model, pretrained from scratch on 5B tokens (40% FineWeb-Edu / 60% DCLM-baseline). No finetuning.

Architecturally a stock Llama, so it loads with plain transformers — no trust_remote_code, no custom modelling file:

lm-eval 0.4.12, zero-shot, acc_norm, float32, CUDA. ArithMark-3 via AxiomicLabs' bencharithmark-3.py (float32, default acc_norm).

ArithMark-3 sits at the 25% chance floor, which is the honest number for a 1M-parameter base model that was never trained on arithmetic.

A finetuned variant of this model reaches 40.40 on the public ArithMark-3 set and 35.5% on AxiomicLabs' private memorization test, but drops to 21.5% on their private variety set after the problems are rephrased — so its arithmetic does not generalize across wording, and it was declined for leaderboard ranking on that basis. Those weights and the full write-up are at opencerebral/littlerock-1M-arithmax. This repository is the base model, with no arithmetic finetuning at all.

The 4,096-token vocabulary is a deliberate constraint: at d=96 a stock 50k tokenizer's embedding table alone would be several times the entire parameter budget. The tokenizer is a byte-level BPE with byte fallback (no UNK is possible), fitted on 400k documents sampled from this model's own pretraining mixture.

Data was materialised to a fixed local binary before training (no streaming), with documents EOS-separated, and the token budget enforced exactly at 5,000,000,000.

This is a ~1M-parameter base model. It is a next-token predictor with no instruction tuning and no alignment, it has very limited world knowledge, and it will produce fluent but frequently false text. It exists to measure what a 1M-parameter budget can do.

Sample (prompt in italics): The capital of France is a national world of fashion, a country and many other nationwide nationality. In this case, the public has been banned in the early 17th century…