atrost/climbmix-llama-288m-2p8b-h100-save-test
Small Llama-style causal language model pretrained from scratch on nvidia/Nemotron-ClimbMix.
Training details
- Architecture:
LlamaForCausalLM
- Parameters:
288,554,240
- Context length:
2048
- Target tokens:
524,288
- Actual tokens:
524,288
- Tokens per optimizer step:
262,144
- Optimizer: AdamW
- LR schedule: cosine decay with warmup
- Precision: bfloat16
- Intended hardware: 8x H100
Model config
The model was initialized from the config embedded in train_climbmix_llama_ddp.py.
