SmalLM V0.1

• W&B run: https://wandb.ai/kickitlikeshika-cision/smallLM/runs/gxtldf56?nw=nwuserkickitlikeshika
• Tokenizer: GPT-2 Pre-Trained Tokenizer
• Hellaswag Acc: 0.3778
• Val Loss: 2.99
• Training Loss: 3.03
• Training Data: Fineweb EDU 10B tokens sample
• Training Steps: 19073
• LR: 6e-4
• Seq Len: 1024
• Vocab Size: 50304
• Number of Layers: 32
• Number of Attention Heads: 32
• Embedding Dim: 1024
• Model Params: 450M