This checkpoint is supervised fine-tuned on the first 150000 valid examples from HuggingFaceTB/smol-smoltalk, using the dataset's messages column and assistant-token-only loss masking.

model config

param	value
parameters	~124M
layers	12
hidden size	768
attention heads	12
context length	1024 tokens
vocab size	50257
positional encoding	RoPE
norm	RMSNorm
activation	SwiGLU
tokenizer	GPT-2 tokenizer

training

base checkpoint: mrinaalarora/mrinaal-124m-base-v3-mathmix/model.safetensors
loaded checkpoint: /root/.cache/huggingface/hub/models--mrinaalarora--mrinaal-124m-base-v3-mathmix/snapshots/e060aaee7cb502b9fd74820ea68659826a6cce09/model.safetensors
dataset: HuggingFaceTB/smol-smoltalk
train split examples loaded: 150000
validation split examples loaded: 3000
skipped train rows over context length: 104986
context length: 1024 tokens
epochs: 1.0
optimizer steps at saved checkpoint: 16500
total tokens seen: 72970859
assistant-label tokens seen: 52170383
train loss at saved checkpoint: 1.8399193286895752
validation loss at saved checkpoint: 1.658129612604777
best validation loss during run: 1.658129612604777
final validation loss during run: 1.6978322048981984
hardware: NVIDIA H100

SFT format

Training examples use a simple GPT-2-tokenizer-compatible chat format:

text

<|endoftext|>user: ...
assistant: ...
user: ...
assistant: ...
<|endoftext|>

Labels are masked so that user/prompt/separator tokens use -100, while assistant answer tokens and the final <|endoftext|> target contribute to loss.

train source mix

everyday-conversations: 1254
explore-instruct-rewrite: 1644
openhermes-50k: 25287
self-oss-instruct: 26402
smol-contraints: 19086
smol-magpie-ultra-short: 51258
smol-summarize-20k: 9117
smol-summarize-5k: 1062
smollm-rewrite-30k: 14890

files

model.safetensors — best checkpoint converted from best.pt
run_summary.json — full training run metadata
last.pt was not uploaded; this repo intentionally publishes the best checkpoint only.

loading

python

from safetensors.torch import load_file

state_dict = load_file("model.safetensors")

To use with the original model class, clone the training repo and:

python

from safetensors.torch import load_file
from first_llm_pretrain.model import DecoderOnlyTransformer, ModelConfig

config = ModelConfig(
    vocab_size=50257,
    block_size=1024,
    n_layer=12,
    n_head=12,
    n_embd=768,
)
model = DecoderOnlyTransformer(config)
model.load_state_dict(load_file("model.safetensors"), strict=False)
model.eval()

strict=False is used because the safetensors conversion removes the duplicate lm_head.weight tensor and keeps token_embedding.weight; the original model class ties those weights.

generation note

For instruction-style inference, prompt with:

text

<|endoftext|>user: your instruction here
assistant:

Stop generation when the model emits <|endoftext|>.

mrinaalarora/mrinaal-124m-instruct-v3-mathmix-smoltalk-150k

mrinaal-124m-instruct-v3-mathmix-smoltalk-150k