Views
No views yet
| optimizer | training steps | batch size | schedule | lr | wd | max_norm | peft? | did i sweep for these? |
|---|---|---|---|---|---|---|---|---|
| bnb.optim.adamw 32-bit | 297 | ~80.5k tokens (T^(2/3)) | 10% step warmup, linear decay to 0 | 2e-5 | 0.1 | 1.0 | full training except rank 64 rslora a=r^0.5 on in_proj_qkv, in_proj_b, in_proj_a, q_proj, k_proj, v_proj, o_proj, out_proj, up_proj, down_proj, gate_proj - merged b4 upload | no |