This is a 10-layer decoder-only Transformer trained on the multivision dataset for 1.0 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 2.805624 (perplexity: 6.9916) on the held-out validation set.
Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset.
1Once upon a time, blue and white bus is parked on the side of the road.
2.
3.
4
5
6.
7
8.
9
10A man and a skateboard on the back of a ramp.
11.
12
13.
14
15
16.
17
18.
19
20a man and a red and white
21.
22a
23a person on a skateboard.
24
25
26.
27a
28
29
30a
1A lonely dragon kite flying through the air.
2
3.
4
5
6
7.
8
9.
10
11A young girl in a red shirt is going down the street.
12.
13
14.
15.
16
17
18
19.
20
21
22a
23
24
25
26
27
28a
29
30a
31
32a man and a green and yellow jacket is holding a baby elephant in a pen.
33a couple of
34
35a
1The opposite of boy is in the air.
2
3.
4.
5
6a woman standing next to a dog on a skateboard
7
8a black and white cat is sleeping in front of a window the head of a dog and a girl on a horse.
9
10.
11
12
13.
14.
15a little girl holding a giant toy in her mouth
16.
17a large window.
18a person on a suitcase
19a person on
1The opposite of queen is walking in the city.
2.
3
4.
5A man and a bus is parked on a street.
6.
7
8
9.
10.
11
12.
13
14
15.
16
17
18.
19
20A man is holding a baseball bat.
21
22
23.
24
25
26.
27a man on a horse.
28
29
30
31.
32
33A woman in a field.
34
35.
36
1My name is next to a wall.
2.
3
4a couple of people on a field
5
6
7
8A person on the front of a bus stop.
9
10.
11A man on a city street.
12a bus is pulled up to a city bus stop
13
14a street.
15.
16
17
18
19
20a
21A man holding a surfboard on the back of a vehicle and a man is
12 + 2 is standing in the grass.
2.
3A
4A person riding a surfboard on the water.
5.
6.
7
8
9
10.
11
12
13
14a man is wearing a skateboard in a skateboard and a wave in the background
15.
16.
17
18
19
20.
21
22
23
24A man riding a skateboard down a ramp.
25A woman on the back of a skateboard.
26
27.
1import torch
2import pickle
3import json
4from train import GPT, GPTConfig, Tokenizer
5
6# Load config
7with open('config.json', 'r') as f:
8 config_dict = json.load(f)
9config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__})
10
11# Load model
12model = GPT(config)
13state_dict = torch.load('model.pt', map_location='cpu')['state_dict']
14model.load_state_dict(state_dict)
15model.eval()
16
17# Load tokenizer
18with open('tokenizer.pkl', 'rb') as f:
19 tokenizer = pickle.load(f)
20
21# Generate
22prompt = 'Once upon a time, '
23input_ids = tokenizer.encode(prompt)
24x = torch.tensor([input_ids], dtype=torch.long)
25with torch.no_grad():
26 for _ in range(50):
27 logits = model(x)
28 probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1)
29 next_token = torch.multinomial(probs, num_samples=1)
30 input_ids.append(next_token.item())
31 x = torch.tensor([input_ids], dtype=torch.long)
32print(tokenizer.decode(input_ids))
1model.pt # Model weights
2config.json # Model architecture config
3dataset.txt # Dataset name used for training
4token_bytes.pt # Token byte mappings
5tokenizer.pkl # Trained BPE tokenizer
6tokenizer_config.json # Tokenizer configuration
7training_metrics.json # Training metrics
8README.md # This file
1@misc{autoresearch_multivision_depth10,
2 title={AutoResearch-multivision-depth10},
3 author={Dustin Loring},
4 year={2026},
5 howpublished={\url{https://huggingface.co/quik-models/elated-frost-137}}
6}}
