learning_rate: 0.0018
train_batch_size: 32
eval_batch_size: 32
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.95) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 0.05
num_epochs: 1.0

Training Loss	Epoch	Step	Validation Loss
No log	0	0	6.9752
2.3540	0.0500	1973	2.3841
2.1531	0.1000	3946	2.1744
2.0805	0.1500	5919	2.0891
2.0298	0.2000	7892	2.0497
1.9944	0.2500	9865	2.0164
1.9774	0.3000	11838	1.9913
1.9386	0.3500	13811	1.9674
1.9219	0.4000	15784	1.9453
1.9060	0.4500	17757	1.9251
1.8754	0.5000	19730	1.8996
1.8544	0.5500	21703	1.8780
1.8337	0.6000	23676	1.8502
1.8023	0.6500	25649	1.8250
1.7808	0.7000	27622	1.7986
1.7482	0.7500	29595	1.7729
1.7302	0.8000	31568	1.7482
1.7060	0.8500	33541	1.7268
1.6923	0.9000	35514	1.7119
1.6893	0.9500	37487	1.7042
1.6835	1.0	39459	1.7027

stage-babylm/llama-256-12L-pairwise

llama-256-12L-pairwise