Museko-125M is a 123M parameter language model I trained from scratch. It targets the sub-150M size class, and across the standard small-model benchmarks it currently comes out ahead of the other models in that range.
It is a Llama style decoder pretrained on FineWeb-Edu with a small amount of synthetic arithmetic mixed in. The arithmetic mix is why it does noticeably better on math than most models its size while still holding solid general knowledge and reasoning.
Benchmarks
All numbers are 0-shot, measured with lm-evaluation-harness and the ArithMark 2.0 benchmark.
Benchmark
Museko-125M
Previous best in class
HellaSwag
33.2
30.6
ARC-Easy
54.0
48.6
ARC-Challenge
25.7
25.4
PIQA
63.2
64.5
ArithMark 2.0
74.3
66.7
Benchmark comparison
By the leaderboard averaging (ARC-Easy and ARC-Challenge combined) this comes to about 52.6, ahead of the previous top model at 49.71. It leads on four of the five benchmarks.
Board average during training
How to read these scores
The numbers above are multiple-choice scores, measured the way the Open SLM Leaderboard does it. The model is shown a prompt with a few candidate continuations and is scored on whether it ranks the correct one highest by likelihood. That is the standard setup and every model on the board is measured the same way, but it is not the same thing as free generation.
There is a real gap between the two. On ArithMark the model ranks the correct answer about 75 percent of the time, but when it has to generate the answer on its own it gets closer to 16 percent. That gap is normal for small models and for likelihood based benchmarks in general. So treat Museko as a strong base model for its size that completes text coherently and ranks well on these tasks, rather than one that reliably writes out correct answers by itself. It is best at completion, not at instructions or question answering.
This is a base model, not an instruction or chat model, so use it for text completion rather than commands or question answering. Like most small base models it loops badly under plain greedy decoding, so the config here ships with a repetition penalty and an n-gram block by default and generation stays coherent out of the box. If you want more varied text, turn on sampling:
out = model.generate(**ids, max_new_tokens=120, do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.3)
Data: FineWeb-Edu plus roughly 10% synthetic arithmetic
Optimizer: AdamW, cosine schedule from 6e-4 down to 6e-5, 1,000 step warmup, weight decay 0.1
Effective batch: about 0.52M tokens per step
Notes
Weights are open under Apache 2.0. This release is a from-scratch model and remains a strong general baseline for its size while being particularly good at arithmetic.
