Views
No views yet
| Hyperparameter | Value |
|---|---|
| Epochs | 2 |
| Maximum sequence length | 16,384 |
| Loss | NLL, assistant tokens only |
| Sequence packing | Enabled |
| Per-device train batch size | 22 |
| Per-device evaluation batch size | 8 |
| Training processes | 8 |
| Gradient accumulation steps | 1 |
| Effective global train batch size | 176 |
| Optimizer | AdamW, fused Torch implementation |
| Learning rate | 1e-5 |
| Minimum learning rate | 1e-6 |
| Scheduler | Cosine warmup with minimum learning rate |
| Warmup ratio | 0.05 |
| Weight decay | 0.05 |
| Adam beta 1 | 0.9 |
| Adam beta 2 | 0.999 |
| Adam epsilon | 1e-8 |
| Maximum gradient norm | 1.0 |
| Precision | bfloat16 |
| Gradient checkpointing | Enabled, non-reentrant |
| Attention implementation | FlashAttention 2 |
| Liger kernels | Enabled |
| Distributed strategy | DeepSpeed ZeRO-2, no offload |
| Shuffle dataset | Enabled |
| Seed | 42 |
| Evaluation interval | 250 steps |
| Logging interval | 1 step |
| Save strategy | Every epoch |
| Save limit | 10 checkpoints |
| Dataset preprocessing workers | 80 |