“knowing a few principles frees you from knowing many facts” R. Descartes “the knowledge of certain principles easily compensates the lack of knowledge of certain facts” C. Helvétius
Experimental research on methods of influencing LLM pre-training to achieve lower validation loss. This is the fourth iteration of the LLMagogy project series.
What is LLMagogy?
LLMagogy explores whether it is possible to influence the pre-training process of a language model through various interventions — staged training, regularization techniques, data ordering, and hyperparameter evolution — to achieve better generalization than standard training.
Previous versions:
LLMagogy V1 — theoretical concept of multi-stage pre-training inspired by human learning https://github.com/loftyara/LLMagogy
LLMagogy V2 — experimental validation on a Python documentation dataset (model growth showed 5-25% time savings) https://github.com/loftyara/LLMagogyV2
LLMagogy V3 — failed attempt on TinyStories (results not published)
Key Results
Baseline (NanoGPT, 7.62M params, 3% TinyStories, BF16): val loss = 2.2026 after 5 epochs.
Effective methods found (table):
Method
Improvement
Notes
Stochastic Depth (before training)
~0.8%
prob=0.2, one epoch
Gradient Noise (after baseline)
~1.4%
std=0.00005, one epoch
Learning Rate reduction
~0.6%
LR to 0.0002
Global Warmup
minor
dropout=0.1, 25k iterations
Final synthesis (Stochastic Depth, then Gradient Noise, then LR reduction): val loss = 2.1270 (~3.5% improvement) in just ~2 epochs instead of 5.
Methods that did NOT work: precision evolution, model size growth, hyperparameter evolution (except LR), loss function modification, curriculum learning, warmup during model expansion.
Project Structure
Directory structure:
data/
raw/ - Raw dataset files
processed/ - Processed data
random/ - Randomly split dataset
curriculum/ - Dataset split by various criteria
docs/ - Documentation (PDF)
scripts/ - Data preparation scripts
01_download_dataset.py
02_prepare_text.py
03_split_cumulative.py
04_tokenize_bin.py
src/ - Training scripts
01_train_baseline.py - Step 1: parameter selection
02_train_baseline_detailed.py - Step 2: baseline training
03_train_evolution_precision.py - Step 3: precision evolution
04_train_evolution_structural.py - Step 4: size evolution
05_train_evolution_hyperparams.py - Step 5: hyperparameter evolution
06_train_evolution_loss.py - Step 6: loss function evolution
07_train_evolution_regulation.py - Step 7: regularization methods
08_train_evolution_datasets.py - Step 8: dataset evolution
09_train_evolution_warmup.py - Step 9: size evolution with warmup
10_train_global_warmup.py - Step 10: global warmup
11_train_synthesis.py - Step 11: synthesis of effective methods
Requirements
Python 3.10+
PyTorch (with CUDA support)
RTX 5070 Ti (16 GB VRAM) or equivalent
All computations in bfloat16
cd scripts
python 01_download_dataset.py - Download TinyStories from Hugging Face
python 02_prepare_text.py - Split dataset into parts
python 03_split_cumulative.py - Split by curriculum criteria
python 04_tokenize_bin.py - Tokenize to .bin files
Training
All training scripts are in src/. Run them sequentially or individually.
cd src
python 01_train_baseline.py - Parameter selection (1 epoch per config)
python 02_train_baseline_detailed.py - Full baseline training (5 epochs)
...
python 11_train_synthesis.py - Final synthesis (best result)
