Views
No views yet
| Attribute | Value |
|---|---|
| Parameters | 71.99M |
| Architecture | 8L / 12H / 768D / 2048FF (SwiGLU, RoPE, RMSNorm) |
| Dataset | TinyStories V2 GPT-4 (roneneldan/TinyStories) |
| Context Length | 256 tokens |
| Vocab Size | 10,000 (BPE trained with gigatoken) |
| Final Loss | Train: 1.2496 | Val: 1.2521 |
| Perplexity | ~3.49 |
1import torch
2from safetensors.torch import load_file
3from model import TransformerLM
4from tokenizer import Tokenizer
5
6# Load tokenizer & model
7tok_data = torch.load("tokenizer.pt", map_location="cpu")
8tokenizer = Tokenizer(vocab=tok_data["vocab"], merges=tok_data["merges"], special_tokens=["<|endoftext|>"])
9
10model = TransformerLM(10000, 256, 768, 8, 12, 2048, 10000.0)
11state_dict = load_file("model.safetensors", device="cpu")
12model.load_state_dict(state_dict)