Continued from the SFT model with a 50% synthetic tool-call mix at a lower LR, specialising in emitting <tool_call>expr</tool_call> for arithmetic while retaining chat ability.

~481M parameters, trained from scratch on a single H100 80GB.

Modern components: RMSNorm (pre-norm), RoPE, SwiGLU FFN, grouped-query attention, QK-norm, value embeddings, U-net style skip connections between layer halves, and a final logit softcap (15 * tanh(logits/15)).

Optimizer: Muon (Newton-Schulz orthogonalisation) on hidden 2-D matrices, AdamW on embeddings / lm_head / norms — embeddings are excluded from Muon by identity.

4,000 steps from the SFT model, 50% synthetic tool-call mix, LR 5e-5. Loss 1.753 -> 1.392.

The full tool loop works end to end:

Q: What is 47 * 89?
A: I'll compute 47 * 89.
   <tool_call>47*89</tool_call>
   <tool_result>4183</tool_result>     <- calculator output, injected
   The answer is 4183.<|im_end|>

Run it with toolcall.generate_with_tools(model, enc, prompt, device).

The tool mechanism is reliable — it emits well-formed calls, the calculator executes them, and the model consumes the result and stops. Choosing the right operation on multi-step word problems is not reliable: e.g. "23 boxes of 17 apples" produced <tool_call>23+17</tool_call> instead of 23*17. That is a reasoning limit of a 481M model, not a formatting failure. safe_calc only evaluates arithmetic (+ - * / ** % //) and rejects anything else, so this is a calculator, never code execution.

This is a raw PyTorch checkpoint, not a transformers model. You need model.py from xerneas3318/modern-lm.

GPT-2 BPE with reserved special tokens in the padded vocab region (IDs 50257+): <|im_start|>, <|im_end|>, <pad>, <think>, </think>, <tool_call>, </tool_call>, <tool_result>, </tool_result>. These are untrained during pretraining (they never occur in the corpus) and are learned at SFT.

Trained on ~10B tokens at 481M parameters — roughly compute-optimal for its size, but small by modern standards. Expect factual unreliability and limited reasoning. No safety tuning of any kind. The base models are next-token predictors and are not instruction-following.