Views
No views yet
.npy files.torch.autocast.1MEMGPT/
2├── configs/
3│ └── config.json # Model and training configuration
4│
5├── data/
6│ ├── edu_fineweb/ # Sharded training data
7│ │ ├── train_000001.npy
8│ │ ├── train_000002.npy
9│ │ └── test_000001.npy
10│ ├── hellaswag/
11│ │ └── hellaswag_val.jsonl
12│ └── fineweb.py # Dataset sharding/processing logic
13│
14├── model_core/
15│ ├── __init__.py
16│ ├── attention.py # Self-attention module
17│ ├── model.py # GPT2 model architecture
18│ ├── dataloader.py # DataLoader_1 class
19│ └── training.py # train_nanogpt function
20│
21├── scripts/
22│ ├── train.py # Entry point to start training
23│ ├── evaluate.py # Run evaluation
24│ └── generate.py # Generate text from trained model
25│
26├── evaluation/
27│ ├── __init__.py
28│ ├── hellaswag.py # HellaSwag dataset preparation
29│ └── val_hellaswag.py # HellaSwag scoring function
30│
31├── logs/
32│ ├── log.txt # Training log file
33│ └── model_xxxxx.pt # Checkpoint files
34│
35├── .gitignore
36├── README.md
37├── requirements.txtconfigs/config.json to configure your model and training setup.1{
2 "model": {
3 "block_size": 1024,
4 "vocab_size": 50304,
5 "n_layer": 12,
6 "n_head": 12,
7 "n_embd": 768
8 },
9 "training": {
10 "max_steps": 19073,
11 "log_dir": "log",
12 "total_batch_size": 524288,
13 "B": 64,
14 "T": 1024,
15 "max_lr": 0.0006,
16 "min_lr": 0.00006,
17 "warmup_steps": 715,
18 "weight_decay": 0.1,
19 "learning_rate": 0.0006
20 }
21}python scripts/train.pytrain_nanogpt() from model_core/training.py using the config in configs/config.json.torchrun --nproc_per_node=NUM_GPUS scripts/train.pyNUM_GPUS with the number of GPUs you want to use.python scripts/evaluate.pyhellaswag_val.jsonl file is available under data/hellaswag/.python scripts/generate.pylogs/ directory.pip install -r requirements.txt.npy sharded data is placed under data/edu_fineweb/.logs/.DataLoader_1 handles distributed data loading.bfloat16 autocasting for better training efficiency.