This is a 8-layer decoder-only Transformer trained on the TinyStories (karpathy/tinystories-gpt4-clean) dataset for 0.2 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 1.013770 (perplexity: 2.0192) on the held-out validation set.
Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset.
1Once upon a time, there was a boy named Tim. Tim had a toy bike and found the toy. He was not so happy. He wanted to help Tim.
2One day, Tim saw a big tree with a big ball.
3Sam gave Tim a big, cold
1A lonely dragon's feet in the water.
2"That's not a big tree! This caterpillar! That's my red and your at it!" it back. "This is my game!" I'm not safe," Tom said.
3Mom, but I don
1The opposite of boy is good at this and you!"
2Tom naughty and his mom learned that it is to be careful and not to others.
3The little girl and the little girl. She went to the park and went to play with her toys. Sue
1The opposite of queen is glad. I have some more fun you will be fun."
2Sam and Mia felt happy. They could turn. They had fun and their twins who always wanted to be kind and make them neat. They did not want the hat and a
1My name is Bobby Tophie for Lucy and Jack.
2Junny let Bob went to play with Bella and his new friend, Bella and his friends played together all day at home and the girl was surprised!
3Jill said,
1import torch
2import pickle
3import json
4from train import GPT, GPTConfig, Tokenizer
5
6# Load config
7with open('config.json', 'r') as f:
8 config_dict = json.load(f)
9config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__})
10
11# Load model
12model = GPT(config)
13state_dict = torch.load('model.pt', map_location='cpu')['state_dict']
14model.load_state_dict(state_dict)
15model.eval()
16
17# Load tokenizer
18with open('tokenizer.pkl', 'rb') as f:
19 tokenizer = pickle.load(f)
20
21# Generate
22prompt = 'Once upon a time, '
23input_ids = tokenizer.encode(prompt)
24x = torch.tensor([input_ids], dtype=torch.long)
25with torch.no_grad():
26 for _ in range(50):
27 logits = model(x)
28 probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1)
29 next_token = torch.multinomial(probs, num_samples=1)
30 input_ids.append(next_token.item())
31 x = torch.tensor([input_ids], dtype=torch.long)
32print(tokenizer.decode(input_ids))
1model.pt # Model weights
2config.json # Model architecture config
3dataset.txt # Dataset name used for training
4token_bytes.pt # Token byte mappings
5tokenizer.pkl # Trained BPE tokenizer
6tokenizer_config.json # Tokenizer configuration
7training_metrics.json # Training metrics
8README.md # This file
1@misc{autoresearch_tinystories_depth8,
2 title={AutoResearch-tinystories-depth8},
3 author={Dustin Loring},
4 year={2026},
5 howpublished={\url{https://huggingface.co/quik-models/wandering-thunder-99}}
6}}
