Views
No views yet
mrinaalarora/mrinaal-124m-base-v2.| share | component | dataset | train tokens | validation tokens |
|---|---|---|---|---|
| 41.67% | finemath-4plus | HuggingFaceTB/finemath, subset finemath-4plus | 625M | 12.5M |
| 23.33% | infiwebmath-4plus | HuggingFaceTB/finemath, subset infiwebmath-4plus | 350M | 7M |
| 20% | fineweb-edu-dedup | HuggingFaceTB/smollm-corpus, subset fineweb-edu-dedup | 300M | 6M |
| 10% | dclm-baseline-1.0 | mlfoundations/dclm-baseline-1.0 | 150M | 3M |
| 5% | cosmopedia-v2 | HuggingFaceTB/smollm-corpus, subset cosmopedia-v2 | 75M | 1.5M |
| param | value |
|---|---|
| parameters | 123,551,232 |
| layers | 12 |
| hidden size | 768 |
| attention heads | 12 |
| context length | 1024 tokens |
| vocab size | 50257 |
| positional encoding | RoPE |
| norm | RMSNorm |
| activation | SwiGLU |
| tokenizer | GPT-2 tokenizer |
mrinaalarora/mrinaal-124m-base-v2/model.safetensors/root/.cache/huggingface/hub/models--mrinaalarora--mrinaal-124m-base-v2/snapshots/d94b2e9f829173b824705eca148febac28a4198c/model.safetensors/vol/datasets/cpt_mathmix_gpt2_1p5b_train/vol/datasets/cpt_mathmix_gpt2_30m_valmodel.safetensors — best checkpoint converted from best.ptrun_summary.json — full training run metadatalast.pt was not uploaded; this repo intentionally publishes the best checkpoint only.1from safetensors.torch import load_file
2
3state_dict = load_file("model.safetensors")1from safetensors.torch import load_file
2from first_llm_pretrain.model import DecoderOnlyTransformer, ModelConfig
3
4config = ModelConfig(
5 vocab_size=50257,
6 block_size=1024,
7 n_layer=12,
8 n_head=12,
9 n_embd=768,
10)
11model = DecoderOnlyTransformer(config)
12model.load_state_dict(load_file("model.safetensors"), strict=False)
13model.eval()strict=False is used because the safetensors conversion removes the duplicate lm_head.weight
tensor and keeps token_embedding.weight; the original model class ties those weights.