You can deploy this model and start earning money today!

mrinaalarora/
mrinaal-124m-instruct-smoltalk-50k

Name: mrinaal-124m-instruct-smoltalk-50k
Availability: InStock

HuggingFaceTB/smol-smoltalk

mrinaalarora/mrinaal-124m-base-v2

finetune

mit

Views

No views yet

mrinaal-124m-instruct-smoltalk-50k

124M-parameter decoder-only causal language model instruction-tuned from mrinaalarora/mrinaal-124m-base-v2.

This checkpoint is supervised fine-tuned on the first 50000 valid examples from HuggingFaceTB/smol-smoltalk, using the dataset's messages column and assistant-token-only loss masking.

model config

param	value
parameters	~124M
layers	12
hidden size	768
attention heads	12
context length	1024 tokens
vocab size	50257
positional encoding	RoPE
norm	RMSNorm
activation	SwiGLU
tokenizer	GPT-2 tokenizer

training

base checkpoint: mrinaalarora/mrinaal-124m-base-v2/model.safetensors
loaded checkpoint: /root/.cache/huggingface/hub/models--mrinaalarora--mrinaal-124m-base-v2/snapshots/d94b2e9f829173b824705eca148febac28a4198c/model.safetensors
dataset: HuggingFaceTB/smol-smoltalk
train split examples loaded: 50000
validation split examples loaded: 1000
skipped train rows over context length: 34741
context length: 1024 tokens
epochs: 1.0
optimizer steps at saved checkpoint: 5500
total tokens seen: 24343600
assistant-label tokens seen: 17409411
train loss at saved checkpoint: 1.8008244037628174
validation loss at saved checkpoint: 1.7162978529930115
best validation loss during run: 1.7162978529930115
final validation loss during run: 1.9427031993865966
hardware: NVIDIA H100

SFT format

Training examples use a simple GPT-2-tokenizer-compatible chat format:

text

<|endoftext|>user: ...
assistant: ...
user: ...
assistant: ...
<|endoftext|>

Labels are masked so that user/prompt/separator tokens use -100, while assistant answer tokens and the final <|endoftext|> target contribute to loss.

train source mix

everyday-conversations: 406
explore-instruct-rewrite: 572
openhermes-50k: 8609
self-oss-instruct: 8716
smol-contraints: 6316
smol-magpie-ultra-short: 17067
smol-summarize-20k: 2997
smol-summarize-5k: 339
smollm-rewrite-30k: 4978

files

model.safetensors — best checkpoint converted from best.pt
run_summary.json — full training run metadata
last.pt was not uploaded; this repo intentionally publishes the best checkpoint only.

loading

python

from safetensors.torch import load_file

state_dict = load_file("model.safetensors")

To use with the original model class, clone the training repo and:

python

from safetensors.torch import load_file
from first_llm_pretrain.model import DecoderOnlyTransformer, ModelConfig

config = ModelConfig(
    vocab_size=50257,
    block_size=1024,
    n_layer=12,
    n_head=12,
    n_embd=768,
)
model = DecoderOnlyTransformer(config)
model.load_state_dict(load_file("model.safetensors"), strict=False)
model.eval()

strict=False is used because the safetensors conversion removes the duplicate lm_head.weight tensor and keeps token_embedding.weight; the original model class ties those weights.

generation note

For instruction-style inference, prompt with: